System and method for detecting and protecting pedestrians

ABSTRACT

System and method for reacting to an impact involving a motor vehicle in which an anticipatory sensor system assesses the probable severity of the impact based on data obtained prior to the impact and initiates deployment of an external safety device via an actuator in the event an impact above a threshold probable severity is assessed. The anticipatory sensor system includes receivers for receiving waves or energy and a pattern recognition system for analyzing the received waves or energy, or data representative thereof, to assess the probable severity of the impact. The pattern recognition system ascertains the identity of an object from which the waves or energy have been emitted, reflected or generated. The pattern recognition system includes a processor embodying a pattern recognition algorithm designed to provide an output of one of a number of pre-determined identities of the object.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of (CIP) of U.S. patentapplication Ser. No. 11/183,598 filed Jul. 18, 2005, now U.S. Pat. No.7,359,782, which is:

1) a CIP of U.S. patent application Ser. No. 11/111,474 filed Apr. 21,2005, now U.S. Pat. No. 7,209,221, which is:

-   -   A) a CIP of U.S. patent application Ser. No. 10/754,014 filed        Jan. 8, 2004, now U.S. Pat. No. 6,885,968, which claims priority        under 35 U.S.C. §119(e) of U.S. provisional patent application        Ser. No. 60/442,204 filed Jan. 24, 2003, and is a CIP of U.S.        patent application Ser. No. 09/851,362 filed May 8, 2001 now        U.S. Pat. No. 7,049,945 which claims priority under 35 U.S.C.        §119(e) of U.S. provisional patent application Ser. No        60/202,424 filed May 8, 2000, and    -   B) a CIP of U.S. patent application Ser. No. 10/180,466 filed        Jun. 26, 2002, now U.S. Pat. No. 6,918,459, which is a CIP of        U.S. patent application Ser. No. 10/097,082 filed Mar. 13, 2002,        now U.S. Pat. No. 6,755,273; and

2) a CIP of U.S. patent application Ser. No. 10/180,466 filed Jun. 26,2002, now U.S. Pat. No. 6,918,459, which is a CIP of U.S. patentapplication Ser. No. 10/097,082 filed Mar. 13, 2002, now U.S. Pat. No.6,755,273, which is a CIP of U.S. patent application Ser. No. 09/825,173filed Apr. 3, 2001, now U.S. Pat. No. 6,623,033, which is:

-   -   A) a CIP of U.S. patent application Ser. No. 09/024,085 filed        Feb. 17, 1998, now U.S. Pat. No. 6,209,909, which is a CIP of        U.S. patent application Ser. No. 08/247,760 filed May 23, 1994,        now abandoned; and    -   B) a CIP of U.S. patent application Ser. No. 09/307,883 filed        May 10, 1999, now U.S. Pat. No. 6,343,810, which is also a CIP        of the '085 application filed Feb. 17, 1998, now U.S. Pat.        6,209,909. These applications are incorporated by reference        herein in their entirety.

FIELD OF THE INVENTION

The present invention relates generally to systems and methods fordetecting and obtaining information about objects around a vehicle whichare likely to impact the vehicle and taking action to reduce thepotential harm caused by the impact. More particularly, the presentinvention relates to anticipatory sensing of an impact involving avehicle and taking action to reduce the potential harm caused by theimpact, whether it is deployment of an internal airbag to protect avehicular occupant, deployment of an external airbag to protect apedestrian or another type of action which can reduce the severity ofthe impact on the vehicular occupants or even possibly avoid the impact.

All of the publications, references, patents and patent applicationsmentioned or referred to herein are incorporated by reference herein intheir entirety as if they had each been set forth herein in full. Notethat this application is one in a series of applications covering safetyand other systems for vehicles and other uses. The disclosure hereingoes beyond that needed to support the claims of the particularinvention set forth herein. This is not to be construed that theinventor is thereby releasing the unclaimed disclosure and subjectmatter into the public domain. Rather, it is intended that patentapplications have been or will be filed to cover all of the subjectmatter disclosed below and in the current assignee's and IntelligentTechnologies International's granted and pending applications. Also,note that the terms frequently used below “the invention” or “thisinvention” is not meant to be construed that there is only one inventionbeing discussed. Instead, when the terms “the invention” or “thisinvention” are used, it is referring to the particular invention beingdiscussed in the paragraph where the term is used.

There are numerous methods and components described and disclosedherein. Many combinations of these methods and components are describedbut in order to conserve space the inventor has not described allcombinations and permutations of these methods and components, however,the inventor intends that each and every such combination andpermutation is an invention to be considered disclosed by thisdisclosure. The inventor further intends to file continuation andcontinuation-in-part applications to cover many of these combinationsand permutations.

BACKGROUND OF THE INVENTION

1. Vehicle Exterior Monitoring

1.1 General

During the process of operating a motor vehicle, it is necessary for theoperator to obtain information concerning the proximity of variousdangerous objects and their relative velocities for the operator to makesound driving decisions, such as whether or not there is enough time tochange lanes. This information should be obtained from the area thatcompletely surrounds the vehicle. In order to gather this information,the operator is frequently required to physically turn his or her headto check for occupancy of a blind spot, for example. In taking such anaction, the attention of the driver is invariably momentarily divertedfrom control of the vehicle.

For an automobile, the blind spots typically occur on either side of thevehicle starting approximately at the position of the driver andextending backwards sometimes beyond the rear of the vehicle. Thelocations of these blind spots depend heavily on the adjustment of theangle of the rear view mirror. Different areas are in the blind spotdepending on the mirror angle. Since it is in general not known whetheror how the mirror is set for the particular vehicle, a blind spotdetector must detect objects anywhere along the sides of the vehicle,and even behind the vehicle, regardless of the mirror setting.

The problem is more complicated for trucks, enclosed farm tractors andconstruction equipment that not only can have much larger blind spotsalong the sides of the vehicle but also can have a serious blind spotstarting in front of the right front bumper of the vehicle and extendingbeyond the right door. This blind spot is particularly serious withtrucks and even vans, SUVs and cars in urban driving where smallvehicles, motorcycles, pedestrians, bicycles etc. in this area can becompletely hidden from the view of the driver.

Several systems have been designed which attempt to rotate the mirror topick up or allow a driver to visually see the object in the blind spot.This is difficult to do without knowledge of the location of the eyes ofthe driver. For most systems that do not incorporate an occupant sensorcapable of determining the location of the driver's eyes, there is arisk that the mirrors will be positioned wrongly thus exacerbatingrather than helping the blind spot detection problem. Also, a systemthat rotates the mirror will make the driver nervous since he or shewill not be able to see the scene that he or she is accustomed to seeingin the mirror.

Monitoring systems that are based on radar or ultrasound have beenavailable but not widely adopted for automobile blind spot detection forreasons related to cost, accuracy and false alarms. Both systems usebeams of energy that can become several feet in diameter by the timethey reach the edges of the blind spot and thus can confuse a largevehicle or a guardrail, sign, parked car etc. two lanes over with avehicle in the blind spot. Some such systems attempt to filterthreatening objects from non-threatening objects based on the relativespeed of the object and thus err by eliminating a significant number ofsuch threats. A tradeoff exists in all such systems where, if allthreatening objects are made known to the driver, the false alarm ratebecomes unacceptable and the driver soon loses confidence in the systemand ignores it. If the false alarm rate is kept low, many dangeroussituations can be ignored.

These prior art systems thus have serious failure modes. The lesson isthat if a vision-based system such as the rear view mirror is going tobe replaced with a non-vision system, then the non-vision system must bealmost as good as the vision system or it will not be adopted.

Some other problems arise when a vehicle strays into the lane of thehost vehicle, i.e., the vehicle with the blind spot detector. Mostsystems will fail to warn the operator and thus an accident can result.As such, the blind spot problem is really two problems relating to themotion of the potentially striking vehicle and the potentially struckvehicle.

A problem that is addressed herein is to determine what information isneeded about the object in the blind spot and then the manner in whichthis information is presented to the vehicle operator so as to eliminateaccidents caused by the failure of the operator to see such an object.This information includes the accurate location of the object relativeto the host vehicle, its size, its relative and/or absolute velocity,and the identity or kind of object. This information must be knownregardless of the changes in road geometry such as steep hills and sharpcurves or changes in environmental conditions. Naturally, the systemmust be low cost if it is going to be purchased by the public orinstalled by vehicle manufacturers.

Studies have shown that giving the driver an extra half-second couldeliminate as many as 50 percent of the accidents. Thus, the risk of anaccident must also be communicated to the operator in a timely fashionto permit the driver to take evasive action or not take a particularaction such as a lane change.

What is needed therefore is a system that acts like the eyes of thedriver and interprets the situation and only gives a warning when thereis a real possibility of an accident. A passive warning can be given inthe form of a light on the mirror whenever an object is in the blindspot; however, an active signal such as an audible signal or anintervention in the steering of the automobile should only be providedwhen it is necessary to prevent an accident. This system must work withvery high reliability and accuracy since the consequences of an errorcan be serious injuries or death.

One approach reported in the paper C. Thorpe et al., “Driving intraffic: Short range sensing for Urban Collision Avoidance”, CarnegieMellon University (CMU), January 2002, is a relatively superficialdiscussion based on the use of vision, radar, ladar and other systemsfor interrogating the environment around the vehicle. No mention is madeof how various objects are identified that could pose threats forvehicular occupants or pedestrians or the distinction between objectsthat may temporarily occupy a space from those that are permanently partof the infrastructure. The technology described below and in otherpatents assigned to Automotive Technologies International (ATI) andIntelligent Technologies International (ITI) presents in detail how suchobjects are found and identified and how the location of fixed objects,such as curbs, are known and are part of a vehicle resident accuratemap. The CMU solution, based on observations without the aid of locationdetermining technologies such as DGPS and accurate maps, will requirereliance on models of vehicles, models of pedestrians and human factors,the analysis of all of which is at best inexact and incapable of solvingthe problem of collision avoidance. Also, by not having a goodidentification of such objects, such as provided herein, the CMUsolution will not be able to provide the proper response in criticalsituations. The inventions described below and in the related patentsand patent applications of ATI and ITI, on the other hand, solve thetotal problem of avoiding fatalities on roadways and lead toward ZeroFatalities. The system and method for achieving this objective isreferred to with the trademarks ZERO FATALITIES™, ROAD TO ZEROFATALITIES® and RTZF®.

The use of range gating is a significant part of several implementationsof inventions disclosed herein. The concept of range gating is currentlynot new when used to determine to distance to a point on an object thatis illuminated by radar, for example (see U.S. Pat. No. 3,735,398 whichdescribes use of range gating with an ultrawide band radar system). Theuse of range gating in conjunction with acquiring an image of an objectand allowing separation or segmentation of the object's image fromreflections from other objects that are at different distances from avehicle is believed to be unique to the inventions disclosed herein.

1.2 Blind Spot Detection Systems

The term “blind spot” as used herein is meant to include more than thecommon definition of the term. See section 8. Definitions for a morecomplete definition.

In U. Dravidam and S. Tosunoglu, “A survey on automobile collisionavoidance system”, Florida conference on recent advances in robotics1999, the authors provide a good review of the field of obstaclesensors. What follows is a summary of their analysis. Obstacle sensorssuch as used for blind spot detection can be divided into three types:

Optical sensors include passive infrared, laser radar and vision. Theygenerally are sensitive to external environmental conditions, which maynot be a problem for blind spot detection since the objects to bedetected are usually nearby the host vehicle. Passive infrared andvision cannot provide a direct measurement of distance to an objectunless part of the field of view is illuminated by a point or structuredlight. Laser radar does provide the capability of direct distancemeasurement, as will be described below, and a stereo camera can alsoprovide distance information.

AMCW (amplitude modulated continuous wave), FMCW (frequency modulatedcontinuous wave) and impulse and noise or pseudo-noise (CDMA—codemodulated multiple access) radar are not generally affected by adverseenvironmental conditions. Although relatively expensive, FMCW radar is agood technique for long-range distance measurement provided the objectto be measured can be separated from other objects. Radar in general hasa high false alarm rate due to the large pixel size at any significantdistance from the host vehicle, to multipath effects and reflectionsfrom signs, bridges, guardrails etc.

Ultrasonics are good in applications where only short relative distancemeasurements are required, since they are able to provide high distanceto the target resolution for a relatively low cost. However, for imagingapplications, the slow speed and relatively large pixel size rendersultrasonics marginal even for close up targets. Also, ultrasonic wavescan be significantly distorted by thermal gradients and wind.

Various researchers have attempted combinations of these technologieswith the particular combination of laser radar and pulse or FMCW beingquite advantageous for long distance collision avoidance applications.

What follows is a brief description of the principles of operation fordifferent types of sensors including their main advantages anddisadvantages. For blind spot applications, sensors should be able toaccurately determine the location of the object and the speed of theobstacle relative to the host vehicle. How well this is achieved can bemeasured with the following indicators:

Sensing range: the maximum and minimum range over which the techniquecan be used.

Range Resolution: the relative change in range that can be measured.

Pixel Resolution: the width of the beam or size of the pixel receivedand to which the sensor is sensitive.

Response time: how quickly the sensor can respond to a change in theblind spot occupancy.

Ultrasonics: These sensors work by measuring the time-to-flight of ashort burst of ultrasound energy typically at a frequency of 30-200 kHz.The time taken for the ultrasonic waves to travel to and return from theobstacle is directly proportional to the distance between the obstacleand the host vehicle. The main advantage is their relative low cost andsmall size. These sensors are also very sensitive to changes in thedensity of air that can be caused by, e.g., high wind velocity andtemperature gradients. Velocity can be measured by the Doppler frequency

Passive Infrared: These sensors measure the thermal energy emitted byobjects. Their main advantage is their low cost and small size, and maindisadvantage is their inability to determine the distance to a detectedobject and slow response time.

Laser Radar: As with regular radar, two techniques exist: (1) apulsed-beam of infrared light coupled with time-of-flight measurements,and (2) the modulation of a continuous light beam. The pulsed techniqueoffers long range, high directionality, and fast response time. Itslimitations are its sensitivity to environmental conditions.

FMCW or AMCW Radar: This type of radar uses modulated microwave ormillimeter frequencies, so that the frequency difference between thereflected signal and the transmitted signal is proportional to therelative velocity of the object. When two waves of slightly differentfrequencies are used, the distance to the object can also be determinedby the phase relationship between the two received reflections. Despiteits high cost, this technique offers the advantages of being insensitiveto environmental conditions, but the disadvantage of having a largepixel size. Velocity can be measured by the Doppler frequency shift.

Impulse Radar: This radar differs from FMCW in that it uses very shortpulses instead of a continuous wave. Like FMCW radar, it is insensitiveto environmental conditions, and the cost is significantly lower thanFMCW. Distance can be determined by time-of-flight measurements andvelocity can be determined from successive distance measurements. Italso has the disadvantage of having a large pixel size resulting in ahigh false alarm rate and too little information to permit objectidentification.

Capacitive and Magnetic: Capacitive and magnetic sensors are able todetect close objects (within about 2 m.), using the capacitance ormagnetic field variations between electrodes excited at low frequencies,typically about 5 kHz. Despite their limited range, they are low incost, and robust to external environmental effects. Poor resolutioncompared to other techniques makes it unlikely that these devices willbe used for blind spot detection since most objects are close to thevehicle.

Vision Systems: These techniques are based on the use of a camera andimage-processing software. They are sensitive to external environmentalconditions; however, this is not a significant shortcoming for blindspot detection. Active infrared vision systems can have significantlylonger range in smoke, fog, snow and rain than human eyesight. This isespecially the case if range gating is used (see U.S. patent applicationSer. No. 11/034,325 filed Jan. 12, 2005, now U.S. Pat. No. 7,202,776).

Considering now some relevant patent prior art. U.S. Pat. No. 4,766,421;U.S. Pat. No. 4,926,170; U.S. Pat. No. 5,122,796; U.S. Pat. No.5,311,012; U.S. Pat. No. 5,122,796; U.S. Pat. No. 5,354,983; U.S. Pat.No. 5,418,359; U.S. Pat. No. 5,463,384 and U.S. Pat. No. 5,675,326 andInternational Publication No. WO 90/13103 are all assigned toAuto-Sense, Ltd., Denver, Colo. and describe modulated optical systems.However, these references do not disclose a camera and in fact, eachreceiver is a single pixel device. The sensor is not mounted on the siderear view mirror but instead is mounted on the rear of the vehicle.These references disclose the use of multiple detectors and therebyachieving a sort of mapping of the detected object into one of severalzones. The references also provide a crude velocity measurement of theobject moving from one zone to another. Otherwise, they do not provideaccurate ranging.

These references describe a blind spot detection system wherein beams ofinfrared radiation are sent from the interrogating or host vehicle at asignificant angle in order to illuminate possible objects in an adjacentlane. No direct measurement of the distance is achieved, however, insome cases multiple detectors are used in such a way that when theadjacent detected vehicle is very close to the detector, that is, belowthe threshold distance, the sensing of the adjacent vehicle issuppressed. In other cases, multiple beams of infrared are used anddistance is inferred by the reception of reflected radiation. Thedetectors are single pixel devices. No attempt is made to image thedetected object. Also, no attempt is made to directly measure thelocation of the detected object.

U.S. Pat. No. 5,008,678 describes a phased array radar system whereinthe antenna can be made to conform to the geometry of an edge of theautomobile. The locations of the antenna, however, make it difficult todetect many objects in the side blind spots. The particular location andvelocity of such objects are also not accurately determined. No image ofthe device is formed. The device is based on a single pixel having arelatively large size making recognition and identification of theobject impossible.

U.S. Pat. No. 5,087,918 describes the use of a combination of two typesof radar: dual frequency Doppler radars and frequency modulatedcontinuous wave radar (FMCW). The system provides an indication of therange of the object from the vehicle but does not indicate where in aplane perpendicular to the vehicle the object is located and thereforewhether it is a threat or not. Also, the system does not apply patternrecognition so that different types of objects in the blind spot can beidentified. This patent gives a good description of the limitations ofradar systems.

U.S. Pat. No. 5,229,975 describes a method for diagnosing when thesystem is not operating properly by placing an LED outside the vehiclenext to the sensor. This is a single pixel device and thus no imaging orobject recognition is possible. Range is not measured directly butthrough a series of sensors whereby each sensor covers a particularzone. Thus, no accurate range measurement is provided. As the objectmoves in the blind spot area, it is sensed by a variety of the sensorsand the last one to sense it gives a crude indication of the distance.

U.S. Pat. No. 5,235,316 describes an ultrasonic blind spot detectingsystem that in fact interrogates as much as 200 degrees around thevehicle. It is mounted in place of the conventional mirror and a newside mirror is provided. The ultrasonic sensor rotates until it locatesan object and then it causes the mirror to rotate so that the driver cansee the object. The patent does not take an image of the threateningobject or the object in blind spot. It is a one-pixel device and it doesnot employ pattern recognition. Additionally, it provides too muchinformation for the driver thus creating the possibility of driverinformation overload.

U.S. Pat. No. 5,289,321 describes a camera and an LCD display on theinstrument panel. The camera views rearward and the driver sees theimage captured on an LCD. It does not disclose a camera mounted on therear view mirror. The main problem is that the LCD driver-viewing screenis more likely to confuse than to aid the driver due to its poor dynamiclight intensity range and the ability to relate the image to thelocation and velocity of the object in the blind spot.

U.S. Pat. No. 5,291,261 describes illumination ports at an angle withrespect to single pixel receiver ports. Fiber optics are used totransmit the few pixels to a central processing station. There is nodirect ranging. Some crude ranging is accomplished since when the objectis in certain zones where the projected light overlays the receivingfields, the reflected light can be sensed. It requires multiplelocations and cannot be mounted, for example, on the side rearviewmirror.

U.S. Pat. No. 5,325,096 uses Doppler radar to determine the presence andrelative velocity of an object blind spot. It filters out stationaryobjects and concentrates only on those objects that have approximatelythe same velocity as the vehicle. As a result, many objects, such as ahigh speed passing vehicle, are missed. A light is used to indicate thepresence of an occupying item in the blind spot area and an audiblealarm is sounded when the turn signal is activated. There is some cruderange measurement possible. It is also a single pixel device and thus,no image of the object can be formed. It invariably will miss objectsthat move rapidly into blind spot. There is no precise ranging. It doesnot appear that the system can be easily adjusted for vehicles ofdifferent length.

U.S. Pat. No. 5,424,952 describes an optical system using cameraswherein distance is measured stereoscopically. Objects that are not inthe adjacent lane are ignored. The problems are that no attempt is madeto analyze the image or to determine its velocity and therefore, a highfalse alarm rate can be expected. Although the image is captured, theinformation is ignored except for its use to determine a stereodistance.

U.S. Pat. No. 5,467,072 describes a phased array radar system that canscan the blind spot as well as all other areas around vehicle. However,the system does not provide an image and therefore no optical patternrecognition is possible. The 10-degree divergence angle of radarindicates that a single pixel has a diameter of over 3 feet at 20 feetfrom the radar transmitter, which is insufficient resolution todetermine the lane that the threatening vehicle is occupying, especiallyif there is a slight curvature in the road. Such a system is notsufficiently accurate to provide drivers who are attempting to mergeinto adjacent lanes with sufficiently accurate position information topermit a safe merge under heavy traffic without visual contact.Additionally, there is no pattern recognition claimed or even possiblewith this low resolution device.

U.S. Pat. No. 5,517,196 describes a multi-frequency radar system usingDoppler techniques. Stationary objects are filtered out. In fact, thesystem also only looks at objects that are traveling at approximatelythe same speed as the host vehicle. It has a good range of 0.25 to 100feet. Some problems are that this system will interfere with othervehicles having the same system. There appears to be no directmeasurement of the object's position, but it does give a good distanceresolution of 0.55 feet. This patent also contemplates the use ofsteering wheel angle and vehicle speed inputs to the system. Even thoughultrasonic, infrared and radar are disclosed, it is still a single pixelsystem. Once again, the system will invariably miss a high-speed vehiclepassing on either the right or the left since it is limited to a twomile per hour velocity difference between the blind spot object and thehost vehicle. It also appears to be a very expensive system. Anotherpotential problem is that when an especially long truck having thesystem of this patent is turning, the system would pick up the end oftruck and treat it as an object in the blind spot.

U.S. Pat. No. 5,668,539 uses thermal imaging to recognize a car or truckin the blind spot. It uses a vibrating element between the field of viewcontaining the blind spot using three lenses thus giving three differentlocations and a reference field of view that is the road behind thevehicle. One problem with this device is that this system does not knowwhere the infrared rays are coming from. It could be from the sun orfrom reflections from the wrong lane. The slow cycle time preventsaveraging to eliminate errors. At a 60 km per hour passing rate, thevehicle will travel 1.7 m each cycle based on a 10 hertz cycle rate. Thepatent also mentions that the form of the signal that comes from avehicle and the blind spot has high frequency associated with it whereasthe form of the signal from the road does not. This is an alternatemethod of discriminating between a vehicle and the road but one thatstill lacks resolution.

U.S. Pat. No. 5,670,935 describes a camera and a display where theactual images of the vehicle in the blind spot and behind the subjectvehicle are displayed on the visual display. Unfortunately, the variousfigures in the patent that illustrate this phenomenon are not accurateand appear to show that the positions of the vehicles relative to thesubject vehicle can be visually seen which is not the case. Thus, theinvention described in this patent cannot be used for blind spotdetection in the manner described since the relative locations ofvehicles cannot be determined. Also, no attempt has been made toidentify and analyze objects in the blind spot and warn the driver of apending accident. U.S. Pat. No. 5,765,116 describes a system wherein atorque is artificially applied to the steering wheel to keep a driver inthe center of his lane. This is not a blind spot related patent but thissame technique can be used to prevent a driver from attempting to changelanes when there is an object in the blind spot.

U.S. Pat. No. 6,038,496 describes a lane boundary finder. It uses alinear array of LEDs plus a linear CCD with a total of 64 pixels in theCCD array. It can be used for blind spot monitoring, although this isnot the main purpose of this invention. The CCD array suffers from theproblem that, due to its limited dynamic range, it can be overwhelmed bylight from the sun, for example, reflected off a vehicle or othersurface. Since there is only a linear array of only 64 pixels, noinformation as to what is in the blind spot can be obtained. In otherwords, the system knows that something is in the blind spot but does notknow what it is or even accurately where it is. Nevertheless, the use ofthe scanning system disclosed wherein the particular pixel or the beamthat is being activated to create a light on a leading or reflectingsurface is an important addition to the technology and may also be usedwith this invention.

U.S. Pat. No. 6,501,371 describes a method for locating eyes of avehicle driver and locating an object external to the vehicle andadjusting a rear view mirror so that the driver sees the externalobject. All of these ideas are believed to have previously beendisclosed in patents assigned to ATI and ITI.

International Publication No. WO 95/25322 describes a passive infraredblind spot detector that processes infrared waves based on a crude formof pattern recognition. There is no accurate ranging and there willlikely be a high false alarm rate with this system. There is alsosometimes a period when the system is unavailable due to changes inambient conditions such as the start of a rain shower or when thetemperature of the road changes due to shading. It is a one-pixel deviceand therefore does not permit the location of the object in the blindspot to be determined. This device and other similar passive infrareddevices will have trouble distinguishing between a small objects such asa motorcycle which is relatively close to the sensor and larger objectssuch as a truck which are relatively far away, for example two lanesover. As a result, it will likely falsely indicate that a relativelylarge object is within a danger zone when in reality the object is at adistance and does not pose a threat.

International Publication No. WO 99/42856 describes a rear of vehiclemounted blind spot detector based on various radar systems. It has thecapability of tracking multiple targets and of accurately determiningthe ranges to the various targets using range-gating techniques. It doesnot attempt to capture an image of an object in the blind spot ordetermine the identity of such an object and thus many non-threateningobjects will appear to be threatening. Accordingly, the system can beexpected to have a high false alarm rate.

In general, the poor resolution of radar systems requires that they userelative velocity as a filter in order to reduce the false alarm rate.As a result, such systems miss a high-speed vehicle that is in the blindspot and was not observed approaching the blind spot by the driver. Thisis a very common occurrence on European superhighways and in the UnitedStates on two lane roads.

Thus, none of the related art described above discloses a method orapparatus of monitoring the area surrounding a vehicle that analyzes animage of one or more objects that occupy the blind spot, identifyingthem and determining the location and relative velocity of the objectsrelative to the host vehicle in a manner that permits an accuratewarning to be issued to the driver of a potentially dangerous situation.

1.3 Optical methods

Optics can be used in several configurations for monitoring the exteriorof a vehicle. The receiver can be a CCD or CMOS imager, to receive theemitted or reflected light. A laser can either be used in a scanningmode, or, through the use of a lens, a cone or beam of light can becreated which covers a large portion of the object in the blind spot.Alternately, a combination of these techniques can be used such as ascanning beam or an adjustable lens system that converts a laser beam toa converging, constant diameter or expanding illuminator. In theseconfigurations, the light can be accurately controlled to onlyilluminate particular positions of interest on the vehicle. In thescanning mode, the receiver need only comprise a single or a few activeelements while in the case of the cone of light, an array of activeelements is needed. The laser system has one additional significantadvantage in that the distance to the illuminated object can bedetermined as disclosed in U.S. Pat. No. 5,653,462.

In a simpler case, light generated by a non-coherent light emittingdiode (LED) device is used to illuminate a desired area. In this case,the area covered is not as accurately controlled and a larger CCD orCMOS array is required. The cost of CCD and CMOS arrays has droppedsubstantially with the result that this configuration is now acost-effective system for monitoring the blind spot as long as thedistance from the transmitter to the objects is not needed. If greaterdistance is required, then a laser system using modulation and phasedetection or time-of-flight techniques, a stereographic system, afocusing system, a combined ultrasonic and optic system, or a multipleCCD or CMOS array system as described herein, or other equivalentsystems, can be used. In a particular implementation, the illuminatinglight is in the form of a modulated infrared laser light that is scannedin a line that illuminates an object in the blind spot. The reflectedlight is received by a pin or avalanche diode, or equivalent, afterpassing through a narrow frequency band notch or other appropriatefilter. The diode is a single pixel device, although several or an arrayof diodes can be used, but since the direction of the transmitted lightis known, the direction of the reflected light is also known. The phaseof received light is then compared with the transmitted light. Themodulating frequency can be selected so that no more than one wavelengthof light exists within the blind spot area. The location of thereflecting object can then be determined by the phase difference betweenthe transmitted and reflected light. Although the described system usesa line scan, it is also possible to use a two-dimensional scan andthereby obtain a three-dimensional map of the area of interest. This canbe done using a pin or avalanche diode or equivalent as described or thelight can be received by a CMOS array and can be monitored on a pixel bypixel basis in a manner similar to the PMD system described in Schwarte,et. al. “New Powerful Sensory Tool in Automotive Safety Systems Based onPMD-Technology” 4th International Conference—Advanced Microsystems forAutomotive Applications, Apr. 6th/7th, 2000, Berlin (Germany). In thislatter case, the entire blind spot area may be flooded with modulatedinfrared light as described in the paper. On the other hand, it isdifficult to overcome the light from natural sources such as the sun bya single floodlight source and therefore a line or even a scanning pointsource permits better distance measurement using a light source ofreasonable intensity. An alternative is to increase the power of thetransmitted illumination, for example by using a high power diode laser,and to increase the beam diameter to remain below eye safety limits.

This technique can also be used for vehicle velocity determination andat the same time the topology of the ground covered by the scanninglaser can be determined and reflections from rocks and other debris canbe eliminated from the velocity calculation for applications where theprime goal is to determine the vehicle velocity relative to the ground.

A mechanical focusing system, such as used on some camera systems candetermine the initial position of an object in the blind spot. Adistance measuring system based of focusing is described in U.S. Pat.No. 5,193,124 (Subbarao) which can either be used with a mechanicalfocusing system or with two cameras. Although the Subbarao patentprovides a good discussion of the camera focusing art, it can be morecomplicated than is needed for the practicing the instant invention. Aneural network or optical correlation system, as described below, canalso be used to perform the distance determination based on the twoimages taken with different camera settings or from two adjacent CCD'sand lens having different properties as the cameras disclosed inSubbarao making this technique practical for the purposes of some of theinventions disclosed herein. Distance can also be determined by thesystem described in U.S. Pat. No. 5,003,166 by the spreading ordefocusing of a pattern of structured light projected onto the object ofinterest. Distance can also be measured by using time-of-flightmeasurements of the electromagnetic waves or by multiple CCD or CMOSarrays as is a principle teaching herein.

There will be conditions when the optical system from the CMOS camerahas deteriorated due to contaminants obscuring the lens. Similarly, thelight emitting laser diodes will emit less light if the lenses aresoiled. The system of this invention contemplates a continuousdiagnostic feature that will permit sensing of either of theseconditions. This can be accomplished in a variety of ways such as alaser diode aimed at the road surface close to the vehicle but withinview of the CMOS camera. If the reflection over a period of time is notsufficient, then a warning light will appear on the instrument panelinforming the driver that maintenance is required. Naturally, there aremany other methods by which a similar diagnostic can be accomplished.

Except as noted, there appears to be no significant prior art for theoptically based apparatus and methods of the inventions disclosedherein. In particular for optical systems that obtain sufficientinformation about objects in the area surrounding the vehicle to permita pattern recognition system.

In U.S. patent application Pub. Nos. 20020191388, 20030193980,20030193981 and 20030198271, a method for illuminating a highway isdisclosed which permits multiple vehicles approaching each other topulse illuminate the roadway in a manner timed by a GPS timing signalsuch that they do not blind each other's imaging system. The imagingsystem is turned off except during an interval necessary for theillumination to travel to the roadway and return. If groups of vehiclestraveling toward each other transmit at different times, when one groupis transmitting the other opposing group has its receiver turned off andthus is not blinded. No mention is made of pattern recognition,positioning the display in the field of view of the driver, or measuringthe distance to the object of interest and thus the limited use of theinventions disclosed in these patent publications is not believed toanticipate inventions disclosed herein.

1.4 Combined Optical and Acoustic Methods

Both laser and non-laser optical systems in general are good atdetermining the location of objects within the two-dimensional plane ofthe image and a pulsed or modulated laser or continuous modulated radarsystem in the scanning mode can determine the distance of each part ofthe image from the receiver by measuring the time-of-flight, correlationor by phase measurement. It is also possible to determine distance withthe non-laser system by focusing as discussed above, orstereographically if two spaced apart receivers are used and, in somecases, the mere location in the field of view can be used to estimatethe position of the object in the blind spot, for example.

Acoustic systems are additionally quite effective at distancemeasurements since the relatively low speed of sound permits simpleelectronic circuits to be designed and minimal microprocessor capabilityis required. If a coordinate system is used where the z-axis is from thetransducer to the object, acoustics are good at measuring z dimensionswhile simple optical systems using a single CCD are good at measuring xand y dimensions. The combination of acoustics and optics, therefore,permits all three measurements to be made from one location with lowcost components as discussed in U.S. Pat. No. 5,835,613 and U.S. Pat.No. 5,845,000.

One example of such a system is an optical system that uses naturallight coupled with a lens and CCD or CMOS array which receives anddisplays the image and an analog to digital converter (ADC), or framegrabber, which digitizes the output of the CCD or CMOS and feeds it toan artificial neural network (ANN), correlation system or other patternrecognition system for analysis. This system uses an ultrasonictransmitter and receiver for measuring the distances to the objectslocated in the area or volume of interest. The receiving transducerfeeds its data into an ADC and from there, the converted data isdirected to the ANN. The same ANN can be used for both systems therebyproviding full three-dimensional data for the ANN to analyze. Thissystem, using low cost components, will permit accurate identificationand distance measurements not possible by either system acting alone. Ifa phased array system is added to the acoustic part of the system, theoptical part can determine the location of the object and the phasedarray can direct a narrow beam to the location and determine thedistance to the object through time-of-flight, for example. Thistechnique is especially applicable for objects near the host vehicle.The combination of radar and optics can also be used in a similar mannerat a significant cost penalty.

Although the use of ultrasound for distance measurement has manyadvantages, it also has some drawbacks. First, the speed of sound limitsthe rate at which the position of the object can be updated. Second,ultrasound waves are diffracted by changes in air density that can occurwhen thermal gradients are present or when there is a high-speed flow ofair past the transducer, compensation techniques exist as reported inthe current assignee's patents and applications such as U.S. Pat. No.6,279,946, U.S. Pat. No. 6,517,107 and U.S. Pat. No. 6,856,876. Third,the resolution of ultrasound is limited by its wavelength and by thetransducers, which are high Q tuned devices. Typically, the resolutionof ultrasound is on the order of about 2 to 3 inches. Finally, thefields from ultrasonic transducers are difficult to control so thatreflections from unwanted objects or surfaces add noise to the data. Inspite of these drawbacks, ultrasound is a fine solution in someapplications such as for velocity and displacement determination forautomobiles in rear end impacts and farm tractors and constructionmachines where the operating speeds are low compared with automobiles.

1.5 Discussion of the External Monitoring Problem and Solutions

The above review of related art blind spot detecting systems illustratesthat no existing system is believed to be sufficiently adequate. Afundamental problem is that vehicle operators are familiar with visualsystems and inherently distrust all other technology. As soon as thenon-visual system gives a false alarm or fails to detect an object inthe blind spot, the operator will cease to depend on the system.Theoretically, the best systems would be based on cameras that allow theoperator to view all of the blind spots. However, there are no adequatedisplay systems that will appear to the operator to be equivalent to anactual view of the scene. CRTs and LCDs require driver concentration anddo not have the dynamic range of lighting that is comparable to the realworld. Either the display will be too bright at night or too dim duringdaylight or the wrong object will be bright compared with the object ofinterest. Although radar systems can accurately measure distance to anobject, they are poor at placing the object in the lateral and verticalcoordinates relative to the vehicle and thus create many false alarms.

The simplest system must be able to accurately position the object inthe blind spot, or other area of interest, relative to the host vehicleand inform the driver that a collision potential exists if the driverdecides to change lanes, for example. This warning must be given to thedriver either at a place where he can almost subconsciously observe thewarning when he is contemplating a lane change maneuver, or it mustprovide an audible warning if he attempts to make such a lane changemaneuver. Finally, such a system might even prevent a driver fromexecuting such a maneuver. A more sophisticated system involving iconswill be discussed below.

To accomplish these goals, it is desirable to positively locate anobject in the area of interest such as one or more blind spots andprovide some identification as to what that object is. A driver willrespond quite differently if the object is a guardrail or a line ofparked cars then he will if it is a Porsche overtaking him at 150 kph.

Thus, the requirements of the system are to identify the object and tolocate the object relative to the host vehicle. To identify the objectpreferably requires a pattern recognition system such as neural networksor optical correlation systems discussed below. To locate the objectpreferably requires some means for measuring the distance from thecamera or other sensor to the object. A CMOS camera is quite capable ofcapturing an image of an object in the blind spot, for example, and ifthe camera is an HDRC camera, then it can perform well under all normallighting conditions from midnight to bright sunshine especially ifminimal illumination is provided on dark nights.

However, even the HDRC camera can be blinded by the sun and thus analternate solution is to use a scanning laser radar where the point ofIR can overpower the emissions of the sun at that wavelength. A scanninglaser radar can scan in either one or two dimensions depending on thedesign. The scanning mechanism can be a rotating polygon mirror,vibrating galvanometer-type mirror, a vibrating MEMS mirror or asolid-state acoustical-optical crystal. In the case of the solid-statedevice, one or more special lenses or reflectors can be used to increasethe effective scan angle. The IR wavelength can be in the far, mid ornear IR bands. If the wavelength is in the mid-IR band, it can beselected so as to provide the greatest range in rain, snow or fog. Also,its amplitude at the selected wavelength should be sufficient to bedetected in bright sunlight. Range gating can also be used to partiallyovercome the effects of rain, snow, fog and/or smoke.

An alternate to the HDRC camera is to use an electronic shutter and/orvariable iris. In this case, the camera can be operated in the rangethat best images objects of interest or a series of images can be takenat different settings and portions of each image combined to create asharp image of the area of interest as reported in S. K. Nayar and T.Mitsunaga, “High Dynamic Range Imaging: Spatially Varying PixelExposures”, Proceedings of IEEE Conference on Computer Vision andPattern Recognition, Hilton Head Island, S.C., June 2000, for example.

The measurement of the distance to the object can be accomplished inmany different ways including ultrasonically, using laser radar (lidar),FMCW or AMCW radar, micropower impulse radar, any of which can becombined with range gating. All of these distance measuring techniquesas well as stereographic, focusing, structured light, triangulation,correlation using random or pseudorandom code modulation and othersimilar techniques are envisioned for use in some of the inventionsdescribed herein and in general there is little prior art describing anyof these methods or systems for monitoring the area exterior to avehicle.

A low-cost preferred approach of solving the distance-measuring problemthat is consistent with an HDRC camera system is to project onto thevolume of the area of interest a series of infrared light pulses. Thesepulses are created by an array of laser diodes that are displaced fromthe camera in such a manner that a pulse of light reflected off of anobject in the blind spot will appear on a certain pixel area in thecamera field of view and since the location of the transmission of thepulse is known and the location of the camera is known, the distance tothe reflecting surface is also known by triangulation. By a judicialchoice of transmission angles from the laser diode array, the entirevolume of the blind spot can be covered with sufficient accuracy so thatno significant object can penetrate the blind spot without creating areflection and thereby permitting the distance to the object to bedetermined. No prior art has been uncovered describing this or a similarprinciple.

In one implementation, a series of pulses from a laser diode array arecontemplated. Other techniques will also accomplish the same goal,however, at a generally higher cost. For example, a continuous laserbeam can be used that would scan the blind spot area, for example, ineither one or two dimensions. Since the direction of the laser will beknown that all times its reflection and excitation of pixels on the CMOSarray would permit, once again, an accurate, mapping of the distance tovarious points on the object in the blind spot to be accomplished. Thistechnique however requires a scanning laser system that in general,although more accurate, would be more expensive than a simple array ofLEDs. Once again, the photonic mixing device described above would alsoprovide a three-dimensional image of the contents of the blind spot aswould a similar and preferred system described below. Another techniqueis to superimpose on the blind spot area, for example, a pattern oflight commonly referred to as structured light. The source of thestructured light must be displaced from the imaging array. By observingcharacteristics of the reflected pattern, such as the distances betweenportions of the pattern, the distance to the object can be determined.This system, although common in machine vision applications, requiresgreater computational resources than the simple LED array describedabove. Nevertheless, it is a viable approach and envisioned for use inthe invention and again there appears to be little, if any prior art forthe use on structured light in monitoring the area surrounding avehicle.

Various forms of structured light coupled with other patterns which areeither inherent in the lens of the camera or are superimposedmathematically on the image can create what is commonly known as Moirépatterns that also permit the determination of the distance from thecamera to the object. In some sophisticated examples, this technique canactually provide the equivalent of topographical maps of the object inthe area of interest that would be of value in interpreting oridentifying an object. However, these techniques require morecomputational power and may are not be as cost-effective as the simpleLED array described above or a linear scanning LED or laser with a pindiode, or equivalent, receiver as disclosed below. Nevertheless, thesetechniques are viable for many applications and will become more so ascomponent prices decrease.

All of these systems permit differentiation between light that isreflected from the transmitted infrared systems and reflected light fromthe sunlight, for example. It is quite likely that at certain times,certain pixels in the camera will receive infrared radiation thatoverwhelms the reflection of the infrared sent by the host vehiclesystem. If this radiation comes from pixels other than those that areexpected, then the system will know that the results are erroneous.Thus, the systems described above have the capability of permitting thediagnosis of the data and thereby achieving a high accuracy of theresults. If the results do not agree with what is expected, then theycan be ignored. If that happens over a significant period of time, thenthe operator of the vehicle is warned that the area monitoring system isnon-operational.

Using sophisticated image processing and mathematical techniques,however, it is expected that those periods of non-functionality will beminimal. The vehicle operator however will not be subjected to a falsealarm but instead will be told that the system is temporarilynon-operational due to excessive sunlight etc. A typical driver caneasily relate to this phenomenon and thereby would not lose confidencein the system. The use of a narrow notch filter, as well as polarizingfilters, can significantly improve the separation of the artificiallyilluminated reflected light from the light reflected from the sun.Additionally, the camera shutter can by synchronized with thetransmitted light.

Initially, one would assume that the only situation that the driver of avehicle should be concerned with is if he or she decides to change lanesand after looking into the rear view mirror and not seeing an object inthe blind spot, proceeds to change lanes. Unfortunately, the blind spotproblem is significantly more complicated. The road may be curved andthe lane changing maneuver might be quite easily accomplished. However,based on the geometry of the blind spot detecting system, using priorart systems, the driver is warned that he cannot execute such a lanechange. This may be fallacious in that the vehicle that the systemdetermines is in the blind spot may actually be in a different lane.Under the stress of congested driving conditions, the driver will nottolerate an erroneous message and thereby he might lose confidence inthe system.

Identification of the object in the blind spot or other area of interestis important and a significant part of one or more of the presentinventions disclosed below. Previous blind spot detectors have onlyindicated that there is a reflection from some object that is near thevehicle that may or may not interfere with the desired intentions of thevehicle operator to change lanes or execute some other maneuver. This isvery disquieting to a vehicle operator who was told that something isthere but not what that something is. For example, let us say that anoperator of a vehicle wished to move that vehicle to the situation wherehe is partially on the shoulder in order to avoid a vehicle that isintruding onto his lane from the right. Most if not all current systemswould tell the vehicle operator that he cannot do so. The systemdescribed in the present invention would say that there is a guard railfifteen feet to your left, thereby allowing movement of 10 feet onto theshoulder and thereby avoid the vehicle intruding onto the lane from theright. This is a real world situation, yet all existing blind spotdetection systems would give an erroneous answer or no answer at all tothe vehicle operator.

Future automobile safety systems will likely be based on differentialGPS and centimeter accurate maps of the roadway. The blind spot detectorof this invention is an interim step to help eliminate some of theaccidents now taking place. The particular geometry of the road isunknown to vehicles today, therefore, a blind spot detection systemcannot use information that says, for example, that the road is about totake a sudden curve to the left, in its decision-making function.Nevertheless, this is a real situation and the system for detectingobjects in the blind spot should not give erroneous information to beoperator that he is about to have collision when the cause of thisanalysis is based on the assumption that the road will be straight whenin fact a strong left turn is taking place. Note that prior to thepresence of accurate maps even the inaccurate maps that now exist or ofprobe vehicle augmented maps can still be used to aid in this problem.

This problem cannot be solved absolutely at this time but if featuressuch as angular position of the steering wheel of the host vehicle aredata that can be entered into the system, then these types of situationscan become less threatening. A preferred implementation of the presentinvention uses data from other vehicle sources in the decision makingprocess including the steering wheel angle, vehicle speed etc. and mapand location information if available.

In prior art blind spot detection systems, the inventor has generallyrealized that the operator of the vehicle cannot be continuouslyinformed that there is an object in the blind spot. Every driver on thehighway during rush hour would otherwise be subjected to a barrage ofsuch warnings. Prior art systems have therefore generally provided anoptical warning typically placed as an LED on the rear view mirror andan audible alert sounded when the driver activates the turn signal.Unfortunately, under normal driving conditions only about 70% of driversuse their turn signals as an indication of a lane change. Understressful congested automobile driving situations, one can expect thatthe percentage would drop significantly. The driver must be warned whenhe is about to change lanes but the activation of a turn signal is notsufficient. Even crude maps that are available on route guidance systemstoday can add valuable information to the system by permitting theanticipation of a curve in the road, for example, especially ifaugmented with data from probe vehicles.

Various studies have shown that the intentions of a driver can sometimesbe forecasted based on his activities during a several second periodprior to execution of the maneuver. Such systems that monitor the driverand, using neural networks for example, try to forecast a driver'saction can be expected to be somewhat successful. However, thesecomputationally intensive systems are probably not practical at thistime.

Another method is to provide a simulated audio rumble strip or vibratingactuation to the steering wheel at such time as the driver elects toredirect the motion of the vehicle based on an object in the blind spot.Whereas a rumble strip-type message can be sent to the driver, controlof the vehicle cannot be assumed by the system since the road in factmay be executing a sharp curve and taking control of the vehicle mightactually cause an accident.

The audio rumble strip method, or a tactile or other haptic messagingsystem, is the preferred approach to informing the driver of apotentially dangerous situation. Initially, a resistance would beapplied to the steering wheel when the driver attempts to alter thecourse of the vehicle. Since the system will not know whether the driveris following a curve in the road or in fact changing lanes, the driverwill be able to easily overcome this added resistance but nevertheless,it should indicate to the driver that there is a potential problem. Ifthe driver persists, then a slight to moderate vibration would beapplied to the steering wheel. Once again, this would be easily overcomeby the driver but nevertheless should serve to positively warn thedriver that he or she is about to execute a maneuver that might resultin an accident based on the fact that there is an object in the blindspot.

The blind spot problem for trucks is particularly difficult. Trucksexperience the same type of blind spot as do automobiles where the blindspot extends the length of the vehicle. However, the truck driver isalso unable to see objects that are in another blind spot extending fromforward of the front of the vehicle back typically 25 feet. This blindspot has been discussed in U.S. Pat. No. 5,463,384 and InternationalPublication No. WO 90/13103. Trucks also have blind spots behind thetrailer that are problematic during backup maneuvers. The inventiondisclosed herein is applicable to all three blind spot situations fortrucks, automobiles or other vehicles.

It is noteworthy that some trucks have the capability of automaticallyrotating the side rear view mirrors based on the relative angle betweenthe cab and the trailer. Such mirror systems are designed so that theymaintain their orientation relative to the trailer rather than the cab.The blind spot monitoring system of this invention can make appropriateuse of this technology to monitor the space along side of the trailerrather then cab.

Buses, trucks and various municipal people conveyors also have a blindspot directly in front vehicle and children have been run over by schoolbuses when the driver was not aware that a child was crossing in frontof the bus after embarking from the bus. The system of this invention isalso applicable for monitoring this blind spot and warning a bus driverthat a child is in this blind spot.

The images obtained from various locations outside of the vehicle canalternately be achieved by cameras or by fiber-optic systems. Theinventions herein are not limited to the physical placement of camerasat particular locations when a fiber optic transmission system could beused as well.

1.6 Lane Departure Warning System

Various vehicle manufacturers are now offering early versions of avision-based lane departure warning system on some vehicle models.

1.7 Night Vision

Various vehicle manufacturers are offering a night vision system on somevehicle models. Some of these systems are based on passive infraredradiation that is naturally emitted from warm bodies and is in the longwave or thermal region of the IR spectrum. Other manufacturers areoffering active IR systems that operate in the near IR region of thespectrum that is just below the visual band in frequency. Despite claimsto the contrary, it is the view of the inventor herein that such systemsare of marginal value and may even contribute to degrading the safety ofthe vehicle since they can act as a distraction (as discussed below).

1.8 Headlight Control

Various systems have appeared from time to time to automatically dim theheadlights of a vehicle when it senses the headlights of an oncomingvehicle. Such systems have now been removed from vehicle models sincethey had a large number of cases where the lights were dimmed when itwas not necessary. Some examples were when the system sensed thereflection of the vehicle's own headlights from a sign, roadwayfurniture on a curve or even from the roadway when the road changes itsangle at the start of a hill, for example. Such systems also did not dimthe lights when the vehicle was following another vehicle.

A more sophisticated approach has been developed as described in U.S.Pat. No. 6,587,573. This patent makes use of pattern recognitiontechniques as disclosed in patents assigned to ATI to determine thatlight sources from vehicle's head or rail lights are present in imagesand are distinguishable from other sources of light. The solutionspresented in this patent are, however, unnecessarily complicated andalternative approaches are disclosed herein.

2. Displays

Several systems have been proposed that display a view of the blindspot, using a video camera, onto a display either on the instrumentpanel or on the windshield as a “heads-up” display. Any system thatdisplays a picture of the object on the screen that is inside thevehicle is also going to confuse the driver since he or she will not beable to relate that picture to an object such as another vehicle in theblind spot on the side of the host vehicle. Additionally, the state ofthe art of such displays does not provide equally observable displays atnight and in bright sunlight. Thus, displays on a CRT or LCD are notnatural and it is difficult for a driver to adjust to these views. Thelighting of the views is too faint when sunlight is present and toobright when the vehicle is operating at night or the brightest object isnot the object of interest and can be difficult to see in the presenceof brighter objects. Therefore, none of the prior art television-likedisplays can replace the actual visual view of the occupant. In thediscussion below, an icon display derivable from a pattern recognitionsystem will be disclosed.

U.S. Pat. No. 6,429,789 discloses the use of icons in a display systemfor sensed exterior objects but does not explain how such objects areidentified or how to effectively display the icons so as to not confusethe driver. To display an icon without knowing which icon to display orwhere to display it is of little value.

U.S. patent application Pub. No. 20040032321 describes use of a camerafor viewing the space behind the vehicle for obstacles and a displayshowing the obstacles, but the display is a video screen and does notdisplay icons. As a result, the display is difficult to see andinterpret.

U.S. Pat. No. 5,949,331 describes a video display with the forecastedpath of the vehicle overlaid. It does not identify other vehicles orobjects nor represent them with icons.

An article on the EE Times website published Jan. 5, 2004, “Head-updisplays get second glance”, describes a multicolor head-up display.

An Article in Nature, Issue 428, pages 911-918, Apr. 29, 2004 titled“The path to ubiquitous and low-cost organic electronic appliances onplastic” provides a good review of the state of the art for organicdisplays based on OLEDs which the inventor herein expects to be thefuture of automotive head-up displays.

European patent application EP1179958 describes use of an overhead view,as well as from any other direction, of a host vehicle along with a viewof the objects that surround the vehicle. This differs from some of theideas disclosed herein in that the invention actually compose a videoimage from up to eight camera images that are mounted on the vehicleexterior. No mention is made of the use of icons and this patent is agood example of the complexity of that approach and of the confusionthat results especially when pixels which are not observable due toblockage are displayed as black. A preferred approach as disclosedherein is to identify objects that are in the vicinity of the hostvehicle and then represent them as icons. This is a far simplercomputational approach, results is a clearer image and allows for thefull representation of the object including pixels that cannot be seenby the cameras. It also permits use of panoramic cameras therebyreducing the total number of cameras to four for imaging all areassurrounding the vehicle.

3. Identification

Neural networks and in particular combination neural networks are usedin several of the implementations of this invention as the patternrecognition technology since it makes the monitoring system accurate,robust, reliable and practical. The resulting algorithm(s) created by aneural network program is usually only a few hundred lines of codewritten in the C or C++ computer language. The resulting systems areeasy to implement at a low cost, making them practical for automotiveapplications. The cost of the CCD and CMOS arrays, for example, havebeen expensive until recently, rendering their use for around vehiclearea monitoring systems impractical. Similarly, the implementation ofthe techniques of the above referenced patents frequently requiresexpensive microprocessors while the implementation with neural networksand similar trainable pattern recognition technologies permits the useof low cost microprocessors typically costing less than $10 in largequantities.

In using neural networks the data is usually preprocessed, as discussedbelow, using various feature extraction techniques. An example of such apattern recognition system using neural networks on sonar signals isdiscussed in two papers by Gorman, R. P. and Sejnowski, T. J. “Analysisof Hidden Units in a Layered Network Trained to Classify Sonar Targets”,Neural Networks, Vol. 1. pp 75-89, 1988, and “Learned Classification ofSonar Targets Using a Massively Parallel Network”, IEEE Transactions onAcoustics, Speech, and Signal Processing, Vol. 36, No. 7, July 1988.Examples of feature extraction techniques can be found in U.S. Pat. No.4,906,940. Examples of other more advanced and efficient patternrecognition techniques can be found in U.S. Pat. No. 5,390,136 and U.S.Pat. No. 5,517,667. Other examples include U.S. Pat. No. 5,235,339, U.S.Pat. No. 5,214,744, U.S. Pat. No. 5,181,254, and U.S. Pat. No.4,881,270.

Although a trained neural network or combination neural network iscontemplated for preferred embodiments of this invention, adaptiveneural networks and other forms of artificial intelligent systems arealso applicable especially where the host vehicle may be towingdifferent loads that could confuse a static trained system. In thiscase, part of the system can be made adaptive to adjust to theparticular load being pulled by the vehicle.

The field of neural networks is a developing field of research and thefuture can be expected to see vast improvements in artificialintelligence systems that go far beyond the concepts now in use ofneural networks and associative memories. All applications of this workto solving automotive safety problems are contemplated herein and theterm neural network will be used herein as representative of the classof such methods now in existence or to be developed. Of particularinterest is a book by Jeff Hawkins, On Intelligence, 2004 Times Books,Henry Holt and Company, LLC, New York, N.Y. This book provides a basisfor the development of future AI methods which will have applicabilityto automotive safety.

4. Anticipatory Sensors

The principles of the inventions disclosed herein can also be used forother purposes such as intelligent cruise control, speed over groundsensors, parking aids, height sensors for active suspensions,anticipatory crash sensors and obstacle detection systems. Oneparticular application is for rear impact anticipatory sensors whereboth the distance and velocity (perhaps using Doppler principles whereapplicable) can be determined and used to deploy a movable headrest orequivalent.

The state of the art as of Apr. 23, 2001 of anticipatory sensors can befound summarized in a report by P. L. J. Morsink titled “Pre-crashsensing for increasing active and passive safety”, TON report)1.OR.BV.013.0/PM which is available on the Internet atpassivesafety.com. Most of the ideas on anticipatory sensing that appearin patents assigned to ATI and ITI covering anticipatory sensing thatpredate this report are not mentioned in the report or are handledsuperficially. Although mention is made of the need for objectclassification and other information, how to do it is generally missingfrom this report. The report lists the following information that isneeded from sensors as:

-   -   time to impact    -   distance to the object    -   object classification (including size and shape)    -   trajectory of the obstacle    -   closing velocity    -   impact direction    -   acceleration of the incoming object    -   mass and stiffness of the incoming object    -   point of impact

One point that is made with regard to anticipatory sensors is “The crashpulse is still necessary because, e.g., a radar sensor cannotdistinguish an empty box or concrete pillar.” Although in principle if aconcrete pillar looks like an empty box, this might be true, however thefrequency of this happening is vanishingly small and in those andsimilar cases, the system would be biased not to set off the airbags forcases where an appropriate pattern recognition system is not sure of theidentity of the object. In such cases, the default position would be torely on the crash sensors. A properly trained pattern recognition systemis not going to confuse a Mack truck traveling toward the vehicle at 60mph with a cardboard box. Thus, the system discussed below will make theproper decision in 99+% of the cases and rely on the crash sensorswhenever it is unable to make a decision. Due to this generalmisconception and teaching away from the inventions disclosed below,plus the 43,000+ roadway fatalities, there is clearly a long felt needfor a proper and accurate anticipatory sensor as disclosed herein.

U.S. patent application Pub. No. 20050065688 also describes some of theideas presented below and in ATI and ITI patents.

4.1 Positioning Airbags

Frontal impacts were the number one killer of vehicle occupants inautomobile accidents with about 16,000 fatalities each year. Sideimpacts were the second cause of automobile related deaths with about8,000 fatalities each year. The number of fatalities in frontal impactsas well as side impacts has been decreasing due to the introduction ofairbags and mandatory seatbelt use laws.

Several automobile manufacturers are now using side impact airbags toattempt to reduce the number of people killed or injured in sideimpacts. The side impact problem is considerably more difficult to solvein this way than the frontal impact problem due to the lack of spacebetween the occupant and the side door and to the significant intrusionof the side door into the passenger compartment which typicallyaccompanies a side impact.

Some understanding of the severity of the side impact problem can beobtained by a comparison with frontal impacts. In the Federal MotorVehicle Safety Standard (FMVSS) 208 49 kph crash test which applies tofrontal impacts, the driver, if unrestrained, will impact the steeringwheel at about 30 kph. With an airbag and a typical energy absorbingsteering column, there is about 40 cm to about 50 cm of combineddeflection of the airbag and steering column to absorb this 30 kphdifference in relative velocity between the driver and vehicle interior.Also, there is usually little intrusion into the passenger compartmentto reduce this available space.

In the FMVSS 214 standard crash for side impacts, the occupant, whetherrestrained or not, is impacted by the intruding vehicle door also atabout 30 kph. In this case, there is only about 10 to 15 cm of spaceavailable for an airbag to absorb the relative velocity between theoccupant and the vehicle interior. In addition, the human body is morevulnerable to side impacts than frontal impacts and there is usuallysignificant intrusion into the passenger compartment. A more detaileddiscussion of side impacts can be found in a paper by Breed et al,“Sensing Side Impacts”, Society of Automotive Engineers Paper No.940651, 1994.

Ideally, an airbag for side impact protection would displace theoccupant away from the intruding vehicle door in an accident and createthe required space for a sufficiently large airbag. Sensors used forside impact airbags, however, usually begin sensing the crash only atthe beginning of the impact at which time there is insufficient timeremaining to move the occupant before he is impacted by the intrudingdoor. Even if the airbag were inflated instantaneously, it is still notpossible to move the occupant to create the desired space withoutcausing serious injury to the occupant. The problem is that the sensorthat starts sensing the crash when the impact has begun is already toolate, i.e., once the sensor detects the crash, it is usually too late toproperly inflate the airbag.

There has been discussion over the years in the vehicular safetycommunity about the use of anticipatory sensors so that the side impactaccident could be sensed before it occurs. Prior to 1994, this was notpractical due to the inability to predict the severity of the accidentprior to the impact. A heavy truck, for example, or a tree is a muchmore severe accident at low velocity than a light vehicle or motorcycleat high velocity. Further, it was not possible to differentiate betweenthese different accidents with a high degree of certainty.

Once a sufficiently large airbag is deployed in a side impact and thedriver displaced away from the door and the steering wheel, he will nolonger be able to control the vehicle that could in itself cause aserious accident. It is critically important, therefore, that such anairbag not be deployed unless there is great certainty that the driverwould otherwise be seriously injured or killed by the side impact.Anticipatory sensors have previously not been used because of theirinability to predict the severity of the accident. As discussed morefully below, the present invention solves this problem and thereforemakes anticipatory sensing practical. This permits side impact airbagsystems that can save a significant percentage of the people who wouldotherwise be killed as well as significantly reducing the number andseverity of injuries. This is accomplished through the use of patternrecognition technologies such as neural networks such as discussed inU.S. Pat. No. 5,829,782.

Neural networks, and more recently modular neural networks, are capableof pattern recognition with a speed, accuracy and efficiency previouslynot possible. It is now possible, for example, to recognize that thefront of a truck or another car is about to impact the side of a vehiclewhen it is one to three meters or more away even in fog, smoke, rain orsnow and further away if range gating is used. This totally changes theside impact strategy since there is now time to inflate a large airbagand push the occupant out of the way of the soon to be intrudingvehicle. Not all side impacts are of sufficient severity to warrant thisaction and therefore, there will usually be a dual inflation system asdescribed below.

Although the main application for anticipatory sensors is in sideimpacts, frontal impact anticipatory sensors can also be used toidentify the impacting object before the crash occurs. Prior to going toa full frontal impact anticipatory sensor system, neural networks can beused to detect many frontal impacts using data in addition to the outputof the normal crash sensing accelerometer. Simple radar or acousticimaging, for example, can be added to current accelerometer basedsystems to give substantially more information about the crash and theimpacting object than possible from the acceleration signal alone.

The side impact anticipatory sensor of this invention can use any of avariety of technologies including optical, radar (including noise radar,Micropower impulse radar, and ultra wideband radar), acoustical,infrared or a combination of these. The sensor system typically containsa neural network processor to make the discrimination however asimulated neural network, a fuzzy logic or other algorithm operating ona microprocessor can also be used.

European Patent Publication No. EP0210079 (Davis) describes, inter alia,a radar system for use in connection with an airbag deployment apparatusto prevent injury to passengers when impact with an approaching objectis imminent. Voltage level inputs representative of the distance betweenan object and the vehicle, the approach rate of the object with respectto the vehicle, the vehicle speed and driving monitor inputs, e.g.,steering angles, turning rates and acceleration/deceleration, are allgenerated by appropriate detectors, weighted according to theirimportance to a normal vehicle operators' sensed safe or danger levelsand then the weighted input voltages are summed to provide an“instantaneous voltage level”. This instantaneous voltage level iscompared with a predetermined voltage level and if the instantaneousvoltage level falls within a predetermined safe zone, output signals arenot produced. On the other hand, if the instantaneous voltage levelfalls outside of the safe zone, i.e., within a danger zone, then thesystem can be designed to initiate deployment of the airbag on theadditional condition that the vehicle speed is above a predeterminedlevel. For example, the system can be programmed to deploy the airbagwhen the vehicle speed is between 35 and 204 miles per hour at a time ofabout 0.2 second prior to impact thereby enabling the airbag sufficienttime to fully inflate.

Davis includes a radar system that includes an antenna assembly, asignal-processing unit and an output monitor. Davis relies on a radarsignal generated by an antenna in the antenna assembly and which causesa return signal to be produced upon reflection of the radar signalagainst the approaching object. The return signal is received by atransceiver to be processed further in order to determine the distancebetween the object and the vehicle and the rate the object isapproaching the vehicle. The return signal from the radar signalgenerated by the antenna is a single pulse, i.e., a single pixel. Theelapsed time between the emission of the radar signal by the antenna andthe receipt of the return signal by the transceiver determines thedistance between the object and the vehicle and based on the elapsedtime for a series of radar signals generated at set intervals, it ispossible to determine the approach rate of the object relative to thevehicle.

In operation, the approach rate of the object relative to the vehicle,the distance between the object and the vehicle, the vehicle speed aswell as other driving parameters are converted to voltage levels. Davisthen uses an algorithm to weigh the voltage levels and compare thevoltage levels to predetermined conditions for which airbag deploymentis desired. If the conditions are satisfied by the results of thealgorithm operating on the weighted voltage levels, then the airbag isdeployed. In one embodiment, by appropriate manipulation of the voltagelevels, false-triggering of the airbag can be prevented for impacts withobjects smaller than a motorcycle, i.e., the voltage corresponding to amotorcycle at a certain distance from the vehicle is smaller than thevoltage corresponding to a truck, for example at that same distance.

Davis does not attempt to recognize any pattern of reflected waves,i.e., a pattern formed from a plurality of waves received over a setperiod of time, from many pixels simultaneously (as with light and CCDs,for example) or of the time series of ultrasonic waves. A tree, forexample can have a smaller radar reflection (lower voltage in Davis)than a motorcycle but would have a different reflected pattern of waves(as detected in the present invention). Thus, in contrast to theinventions described herein, Davis does not identify the object exteriorof the vehicle based on a received pattern of waves unique to thatobject, i.e., each different object will provide a distinct pattern ofreflected or generated waves. The radar system of Davis is incapable ofprocessing a pattern of waves, i.e., a plurality of waves received overa period of time, and based on such pattern, identify the objectexterior of the vehicle. Rather, Davis can only differentiate objectsbased on the intensity of the signal.

International Publication No. WO 86/05149 (Karr et al.) describes adevice to protect passengers in case of a frontal or rear collision. Thedevice includes a measurement device mounted in connection with thevehicle to measure the distance or speed of the vehicle in relation toan object moving into the range of the vehicle, e.g., another vehicle oran obstacle. In the event that prescribed values for the distance and/orrelative speed are not met or exceeded, i.e., which is representative ofa forthcoming crash, a control switch activates the protection andwarning system in the vehicle so that by the time the crash occurs, theprotection and warning system has developed its full protective effect.Karr et al. is limited to frontal crashes and rear crashes and does notappear to even remotely relate to side impacts. Thus, Karr et al. onlyshows the broad concept of anticipatory sensing in conjunction withfrontal and rear crashes.

U.S. Pat. No. 4,966,388 relates to an inflatable system for side impactcrash protection. The system includes a folded, inflatable airbagmounted within a door of the vehicle, an impact sensor also mountedwithin the door and an inflator coupled to the impact sensor and in flowcommunication with the airbag so that upon activation of the inflator bythe impact sensor during a crash, the airbag is inflated.

U.S. Pat. No. 3,741,584 shows a pressurized air container and two airlines leading to a protective air bag. An air line passes through afirst valve which is controlled by an anticipatory sensor and the otherair line passes through a second valve controlled by an impact detector.The purpose of having two sensors associated with different valves is toensure that the protective bag will inflate even if one of the crashsensors does not operate properly.

U.S. Pat. No. 3,861,710 shows an airbag inflation system with a singleairbag which is partially inflated based on a signal from an obstacledetecting sensor and then fully inflated based on a signal from animpact detecting sensor. The obstacle detecting sensor controls releaseof gas from a first gas supply source into the gas bag whereas theimpact detecting sensor controls release of gas from a second gas supplysource into the gas bag. The first gas supply source includes a firstgas container filled with a proper volume of gas for inflating the gasbag to a semi-expanded condition, a first valve mechanism, a pipebetween the first gas container and the first valve mechanism and a pipebetween the first valve mechanism and the gas bag. The second gas supplysource includes a second gas container filled with gas in a volumesupplementing the volume of gas in the first gas container so that thecontents of both gas containers will fully inflate the gas bag, a secondvalve mechanism, a pipe between the second gas container and the secondvalve mechanism and a pipe between the first valve mechanism and the gasbag.

U.S. Pat. No. 3,874,695 shows an inflating arrangement including twoinertia-responsive switches and coupled gas-generators. Thegas-generators are triggered by the switches to inflate an airbag. Theswitches are both crash sensors and measured acceleration producedduring the collision, and thus are not anticipatory sensors. The purposeof the two switches operative to trigger respective gas-generators is toenable the airbag to be inflated to different degrees. For example, ifthe crash involving the vehicle is a low speed crash, then only switchis actuated and gas-generated is triggered and the airbag will beinflated to part of its full capacity.

In U.S. Pat. No. 5,667,246, there are two accelerometers, each of whichprovides a signal when the value of the increase in deceleration exceedsa respective threshold value. The signal from the accelerometer is setto a first ignition stage and through a delay member to a secondignition stage. The second ignition stage also receives as input, asignal from the accelerometer and provides an inflation signal only whenit receives a signal from both accelerometers. In operation, when theaccelerometer sends a signal it serves to partially inflate the airbagwhile full inflation of the airbag is obtained only by input from bothaccelerometers.

Taniguchi (JP 4-293641) describes an apparatus for detecting a bodymoving around another body, such as to detect a car thief moving arounda car. The apparatus includes a detection section supported on a supporttoll to the roof of the car. Taniguchi states that the detection sectionmay be based on an infrared, microwave or ultrasonic sensor.

4.2 Exterior Airbags

Externally deployed airbags have been suggested by Carl Clark andWilliam Young in SAE paper “941051 Airbag Bumpers”, 1994 Society ofAutomotive Engineers. Clark and Young demonstrated the concept ofpre-inflating the airbag for frontal impacts but did not provide amethod of determining when to inflate the airbag. The sensing problemwas left to others to solve.

4.3 Pedestrian Protection

Although numerous patents have now appeared that discuss using anexternal airbag for the protection of pedestrians, little if any priorart exists on using anticipatory sensing for the detection of apedestrian and for the pre-inflating of the pedestrian protectionexterior airbag. Let us now consider some related art. None of therelevant ideas on the detection and classification of objects around thevehicle are believed to predate those disclosed in U.S. Pat. No.6,343,810 and its linked predecessors.

Stereo cameras and neural networks are disclosed in a 1999 paper by L.Zhao and C. Thorpe at the itsc'99 Conference in Tokyo, Japan titled“Stereo- and Neural Network-Based Pedestrian Detection” and earlier inU.S. Pat. No. 6,370,475. The paper mentions that it can operate at videorate (30 frames per second) which is fast enough for automotiveapplication; however, later they say that the stereo system can operateat 3-12 frames per second. Twelve frames would border on being too slow.The detection rate was stated at 85.2% with a false alarm rate of 3.1%.It is difficult to believe that such a system would be used forpedestrian detection for automobiles where hood raising or externalairbags would be deployed with such a high false alarm rate. For suchautomotive applications, the false alarm rate would have to beeffectively zero due to the large number of non-pedestrian scenes that avehicle encounters on every trip. Other concepts discussed in the paperinclude:

-   -   Use of depth (from stereovision) segmentation to select        candidate windows.    -   Normalization of candidate windows to 30×65 pixels.    -   Feed intensity gradient image directly into a feed-forward        neural network.    -   Train using back-propagation and bootstrapping.

Note that the performance is not high even with a small database andthat depth segmentation will not work if pedestrians are not close tothe camera.

T. Evgeniou and T. Poggio, in “Sparse Representation of MultipleSignals”, MIT AI Laboratory, September 1997, A.I. Memo No. 1619, attemptto fine sparse representations of a class of signals specifically thosefrom images of pedestrians. In a sense, they use pattern recognition orit can be thought of as a deterministic determination. In any event, therecommend method is too complicated and too slow for practical use.Also, the accuracy of recognition of a pedestrian is not reported and itonly considers forward facing pedestrians. This paper is related topedestrian recognition only indirectly. It is a pattern-matching relatedapproach, which has similar problems to those that have to be solved forRadial Basis Function (RBF) and Support Vector Machine (SVM) networks toreduce number of memorized support vectors.

This paper is very mathematical and is not closely related to pedestriandetection. The only application in the pedestrian detection problem isto help in the selection of a compact set of features from anover-complete transform such as wavelet decomposition.

C. Wohler, J. K. Anlauf, T. Portner and U. Franke, in “A time delayneural network algorithm for real-time pedestrian detection” 1998 IEEEInternational Conference on Intelligent Vehicles, pages 247-252, discussa motion-based pedestrian detection system using stereo cameras whichappears to also be too slow and too inaccurate. It uses neural networksin a time delay fashion which is similar to the combination neuralnetworks developed by the current assignee. It is based on leg motionand therefore is not viable for pedestrians walking head-on into or awayfrom an oncoming vehicle. It discusses using a fast stereo algorithm andmotion tracking uses a Kalman filter but no data is provided as to howthis is implemented.

C. Papageorgiou, T. Evgeniou and T. Poggio, in “A trainable pedestriandetection system”, Proceedings of Intelligent Vehicles, StuttgartGermany October 1998, discuss a system that appears to not be based on avehicle and even in its most advance form, the system requires 1.2seconds to detect and classify a pedestrian and thus is too slow. Sincethe distance to a pedestrian is not recorded, it is difficult to makeany judgment as to the accuracy of the system. This paper discusses theuse of wavelets, edges and SVM, all of which have limitations asdiscussed herein.

The key point in this approach is feature extraction. Once the featurevectors are extracted, they can be processed using SVM, RBF, or simplefeed-forward neural networks as has been done for years by the currentassignee.

D. M, Gavrila, in “Pedestrian Detection from a moving vehicle”,Proceedings of the European Conference on Computer Vision, pp 37-49,Dublin, 2000, also reports on a pedestrian detection system. This systemrequires 1 second using a dual Pentium 450 megahertz processor. This isclearly too slow and requires too much processing power so as to renderthis system impractical. It makes use of template matching which is aslow computationally intensive pattern recognition method.

The technology in this article and in U.S. Pat. No. 6,556,692 by thesame author focuses on finding the regions of interest or candidatewindows in the images. The method is shape-matching (or correlation)against a preset template tree. The shape is found using edge detection,and fuzziness is added by distance transform or simply low passfiltering. In this paper, it is also proposed to use pattern matchingapproach based on texture features. FIG. 1( d) demonstrates pictureafter texture processing (DT). This is similar to some work done by thecurrent assignee's employees. The inventor herein believes that patternmatching approaches should be avoided as they are very limited byartificially constructed pattern templates.

As discussed in U.S. Pat. No. 6,556,692 and in the current assignee'sprior patents, when using neural networks approaches, “a prioriknowledge” about the objects is not required. It is enough to have aproperly marked training set with sufficient size. Modular neuralnetworks are a better approach than the tree structure used in the '692patent. Also, instead of softening with the distance transformation asdiscussed, a better approach is to use a “stop at an edge” technique,shifts of initial image and fixed set of networks in an ensemble, whichcan simulate “eye saccades”, and allows precisely defined and mostprobable place of an object in the image and increased the probabilityof a successful recognition.

U.S. patent application Pub. No. 20050084156 describes neural networksas something others might do. Since the effective use of neuralnetworks, as taught in the current assignee's patents, is somewhatcomplicated. The fact that others might consider their use forpedestrian detection is not an enabling statement. The techniquediscussed uses a multi-path classification or detection:

-   -   Stereovision: obtain depth map and then run exhaustive search of        the template database.    -   Image feature-based: use neural network to process features such        as edge energy, intensity variation, symmetry, shadow,        histogram, etc.    -   Include other data sources such as radar.        The only detailed approach in this patent application is        (histogram-based, no stereo, and only for separation between car        and pedestrian):    -   1. Generate image histogram    -   2. Find a double threshold using contour score (with an        improbable formula)    -   3. Generate binary image by applying double threshold    -   4. Generate row-sum and column-sum vectors    -   5. Calculate object score and apply threshold (using the same        faulty formula)

N. Checka, in “Fast pedestrian detection from a moving vehicle”,Proceedings of the 2004 Student Oxygen Workshop, MIT Computer Scienceand Artificial Intelligence Laboratory, uses cascade-based neuralnetworks which is a subset of modular neural networks developed by thecurrent assignee. No accuracy or speed data is provided in this paper.This paper is a very brief description. Nothing informative is presentedexcept that the AdaBoost algorithm is used in a cascade architecture ormultistage classification. There is no detailed explanation of theapproach. In this paper some texture-like features appear to be used,actually wavelets, and some sort of modular neural networks. Bycomparison the assignee is using a wider range of features includingedges and contours and has selected better texture features. There isnothing said about using sequences of images which can greatly improvethe accuracy.

While reading Fang, Y., K. Yamada, Y. Ninomiya, B. K. P. Horn, & I.Masaki “A Shape-Independent Method for Pedestrian Detection withFar-Infrared Images,” IEEE Transactions On Vehicular Technology, Vol.53, No. 5, September 2004, it becomes obvious that the authors of thispaper, as well as all of the previous papers discussed above, do notreally understand neural networks. Their application of neural networksis very primitive and doomed to failure. Also, this system is based onthermal infrared technology and it is very difficult to get the distanceto an object based on thermal IR unless stereo cameras are used to whichalso is relatively inaccurate for longer separations. The accuracy ofthe system probably is poor since the distance to the pedestrian is notprovided. At six frames per second, this system is also too slow.

The proposed method applies to night vision only. The detection consistsof two stages: hypothesis and verification (a common approach to manydetection/classification problems). The hypothesis (or segmentation)stage is done as follows:

-   -   1) Binarize far-infrared images    -   2) Count bright pixels at each column    -   3) Find peaks of pixel count along the horizontal direction    -   4) Within the regions of peaks, narrow down the vertical        position based on brightness and intensity variation along the        horizontal direction.        The verification (or classification) stage is done by using        (with or without a combination of) the following methods:    -   Similarity of the intensity histogram.    -   The fact that bright pixels usually locate near the center of        the region of a pedestrian and often locate close to the        boundary of the region of a non-pedestrian.    -   The fact that the pedestrian regions locate more closely in a        data space defined by the vertical edge pixel counts in the        region and its immediate upper/lower neighbors.

A. Shashua, Y. Gdalyahu and G. Hayon. Pedestrian Detection for DrivingAssistance Systems: Single-frame Classification and System LevelPerformance. Proc. of the IEEE Intelligent Vehicles Symposium (IV2004),June 2004, Parma, Italy, discloses a speed of ten hertz which ismarginal. The accuracy is probably poor however since the accuracy iscritically based on the distance from the vehicle and since this is notstated it is difficult on any of these papers to judge what the accuracyis when the pedestrian is no more than ten meters away. The method ofdividing the pedestrian to separate segments is unnecessary if oneunderstands the basis of neural networks and it is probably a poorchoice in any event. Although it is difficult to ascertain, it lookslike that for a distance of up to fifteen meters there is a ninety sixpercent accuracy with some false positives; however, it takes 4.6 framesto reach this accuracy which is approximately one half second whichwould be too slow. In the opinion of the inventor herein, the actualcapability of generalization is not as good as it sounds in this paper,and the flat-road assumption can cause problems in the real world.

To summarize:

-   -   The paper focuses on single-frame classification.    -   The single-frame classification algorithm has 2 stages:        -   1. The region of interest is divided into sub regions, and            orientation histogram (or edge energy) is extracted from            each sub region. Then Ridge Regression is used to calculate            the weights with which the best linear separation can be            achieved within individual training clusters (i.e., subsets            of training set).        -   2. Then, the inner-product of the weights and the feature            values forms a new feature vector and AdaBoost is used to            optimize against the entire training set.    -   The paper lacks details on other modules of the entire system.

The authors mention an “attention mechanism”, which generates about afixed number of candidate windows (75 on average). The current assigneealso employs an “attention mechanism” that generates arbitrary number ofcandidates depending on a particular image but usually the number ismuch smaller. The “attention mechanism” isn't described in the paper soit is difficult to evaluate it. The current assignee uses neuralnetworks inside the “mechanism” to increase probability to find theRegion of Interest (ROI) containing a pedestrian.

Similar to the described approach, the currently assignee also treats“Single Frame Classification Algorithm” (Part III) as one of mostimportant parts of the system. Most important in the presented paper issplitting ROI to 9 sub-regions for independent analysis and manuallypreparing 9 separate training sets. These are strong limitations of thealgorithm because it becomes suitable for pedestrians only. Also theyuse oriented gradients to construct a feature vector for recognition and(as it is described) some texture parameters to find a ROI. Thepresented approach is also limited. The assignee uses both types offeatures for recognition, as well as contours. The authors build theirsystem on the 9 sub-regions instead of complex net for whole ROI. Theymentioned that the results with a complex network, as taught by thecurrent assignee, is better.

Several key features related to pedestrian recognition were firstdisclosed by ATI and ITI in the patents referenced herein. Among theseideas are the placement of the camera on the rear view mirror, the useof pattern recognition which includes template matching, segmentation,use of stereo or multiple cameras for monitoring the area of the spacesurrounding the vehicle, the general concept of detecting andclassifying of pedestrians, vehicles, and other objects surrounding thevehicle, use of the visual, near infrared or far infrared portions ofthe electromagnetic spectrum and various advantages of using each ofthese particular portions of the spectrum.

In particular, the method of segmentation which is how to separateinteresting objects in a scene so that they can be isolated and analyzedand identified separately using stereo vision, a scanning laser radarwith distance measuring capabilities, and range gating were all believedto have been first disclosed by ATI and ITI in their patents. Severalcombinations of these techniques have been disclosed as examples of howone would combine various techniques. For example, a laser radar can beused in conjunction with a night vision far infrared system to measurethe distance to the object emitting the radiation. In fact, using radaror any other method of detecting that there may be an object of interestincluding using the radiation emitted by that object which can be in theform of visual, thermal, or acoustic can be used to get the attention ofsystem and subsequently cause a thin beam scanning laser radar todetermine the location of emitting object. One technique which was notdisclosed in detail but nevertheless obvious is to obtain a roughestimate of the distance of an object, and the object is classified forexample as a pedestrian based on the size of the region of interest thathas been isolated to contain the image of the pedestrian. Other obvioustechniques are based on the vertical location of the object of interestin the camera field of view which may require the assumption that theroad is relatively flat.

4.4 Rear Impact

The first disclosure of any rear impact sensor, either anticipatory or acontact sensor, is believed was made in U.S. Pat. No. 5,629,681 and U.S.Pat. No. 5,694,320. Such sensors can be used as disclosed in the currentassignee's patents for deploying a variety of whiplash protectiondevices and the first disclosure of such a device deployed by a sensorwas also made by the current assignee in the above patents. AlthoughJapanese patent No. 2003-112545 is related, it is believed to follow theinvention by the current assignee's inventors.

4.5 Positioning of Out-of-Position Occupants

U.S. patent application Ser. No. 11/423,596, now U.S. Pat. No.7,338,069, describes positioning airbags and especially for such airbagsthat are triggered by an anticipatory sensor, and any prior art citedtherein might be relevant to this aspect.

5. Agricultural Product Distribution Machines

A description of agricultural product distribution machines is set forthin the parent '598 application and in U.S. Pat. No. 6,285,938.

6. Distance Measurement

As discussed above, regardless of the distance measurement system used,a trained pattern recognition system, as defined below, can be used toidentify and classify, and in some cases to locate, the illuminatedobject. Distance measurements by a variety of techniques can be used todetermine the distance from the sensor to an object in the monitoredarea. They can also determine the distance to the ground foragricultural applications, for example, and provide correctioninformation of the effective angle of transmission as will be discussedbelow.

Use of passive optical camera systems, such as the HDRC camera, has beendiscussed and the method of using either neural networks, opticalcorrelation, or other pattern recognition systems has also been and willbe discussed that illustrates how, in the present invention, theidentity of the object occupying the area of interest will bedetermined. What follows now is a more detailed discussion of positiondetermination.

For a preferred implementation of the system, the light from laserdiodes will cross the field of view of the camera. If there is a heavyfog, for example, in the monitored area, then the reflection of thelight off of the fog will create an elliptical image on the camerasensor array. This would also be true when heavy rain, smoke or heavysnowfall is present. This fact can be used to determine visibility.Observations of visibility conditions of objects in the area surroundingthe vehicle even during severe weather conditions has led the inventorof this invention to the conclusion that when the visibility is so poorthat the optical system using laser diodes described herein, forexample, is not functioning with sufficient accuracy, that the operatorof the vehicle should not be operating the vehicle on the roads andtherefore the vehicle operator should be informed that safe travel isnot possible. Thus, the use of radar or other technologies to view theblind spot, for example, which is actually quite close to the vehicle,is not necessary since vehicle operation should not be permitted whenthe visibility is so poor that the object cannot be seen in the blindspot, for example, by the systems of this invention. Nevertheless, theinventions herein can contribute to safe driving in these conditions, ifsuch driving is attempted, since an indication will be obtained by thesystem based on the elliptical reflections from the laser diodeindicating that the visibility is unacceptable. Note that when using ascanning IR laser radar system, the range of view of the system greatlyexceeds that of the human operator especially when range gating is usedto remove close-up reflections from the atmosphere (rain, snow, fog,smoke etc.)

For the embodiment of the invention using triangulation, it is desirablefor the laser diodes, scanning laser diode or other light source to bedisplaced as far as reasonably possible from the camera in order topermit the maximum accuracy for the triangulation calculations. In anautomobile, as much as six inches exists from one side of the exteriorrear view mirror to the other side. This is marginal. For large trucks,the vertical distance separating the top and bottom of the rear housingcan be as much as 24 inches. In both cases, the laser diode would beplaced at one extreme and the camera at the other extreme of the mirrorhousing. An alternate approach is to place the camera on the mirrorhousing but to place the light source on the vehicle side. Alternately,both the camera and the light source can be placed at appropriatepositions on the side of the vehicle. The key is that the direction ofthe light source should cross field of view of the camera at preferablya 10 degree angle or more.

Since the dots or a line created by a light source used to monitor thearea of interest will likely be in the infrared spectrum and themajority of the light coming from objects in the monitored area will bein the visible spectrum, the possibility exists to separate them throughthe use of an infrared filter which will allow more accurately thedetermination of the location of the reflection from the laser diodeonto the optical array. Such filters can be done either mathematicallyor through the imposition of a physical filter. However, this approachcan require a mechanical mechanism to move the filter in and out of thecamera field of view if visible light reception is also desired.Alternately, to eliminate the need to move the filter, a pin diode orequivalent dedicated receiver can be used to receive the reflectedinfrared light. Of course, multiple imagers can also be used, one forinfrared and another for visible.

7. Scanners

A large number of patents and literature is available on rotating andvibrating mirror scanners and need not be listed here. These includerotating polygons such as used in surveying and office copiers andoscillating mirrors as in galvanometer type approaches. Scanners basedon acousto-optical principles for use in automotive applications are newand are disclosed in the above-referenced patents and patentapplications to ITI. A discussion on acousto-optic scanners can be foundin U.S. Pat. No. 6,560,005.

8. Definitions

Preferred embodiments of the invention are described below and unlessspecifically noted, it is the applicant's intention that the words andphrases in the specification and claims be given the ordinary andaccustomed meaning to those of ordinary skill in the applicable art(s).If the applicant intends any other meaning, he will specifically statehe is applying a special meaning to a word or phrase.

Likewise, applicant's use of the word “function” here is not intended toindicate that the applicant seeks to invoke the special provisions of 35U.S.C. §112, sixth paragraph, to define his invention. To the contrary,if applicant wishes to invoke the provisions of 35 U.S.C. §112, sixthparagraph, to define their invention, he will specifically set forth inthe claims the phrases “means for” or “step for” and a function, withoutalso reciting in that phrase any structure, material or act in supportof the function. Moreover, even if applicant invokes the provisions of35 U.S.C. §112, sixth paragraph, to define his invention, it is theapplicant's intention that his inventions not be limited to the specificstructure, material or acts that are described in the preferredembodiments herein. Rather, if applicant claims his inventions byspecifically invoking the provisions of 35 U.S.C. §112, sixth paragraph,it is nonetheless his intention to cover and include any and allstructure, materials or acts that perform the claimed function, alongwith any and all known or later developed equivalent structures,materials or acts for performing the claimed function.

The definitions set forth in the parent '598 application, section 8 ofthe Background of the Invention section, are applicable herein.

OBJECTS AND SUMMARY OF THE INVENTION

An object of the present invention is to provide a system and method fordetecting and obtaining information about objects around a vehicle whichare likely to impact the vehicle and taking action to reduce thepotential harm caused by the impact.

Additional objects and advantages of the invention, or other disclosedinventions, are set forth in the parent '598 application.

In order to achieve some of these objects, a system for reacting to anexpected impact involving a motor vehicle in accordance with theinvention includes a safety device deployable outside of the vehicle, ananticipatory sensor system for assessing the probable severity of animpact involving the vehicle based on data obtained prior to the impactand initiating deployment of the safety device in the event an impactabove a threshold probable severity is assessed and an actuator coupledto the anticipatory sensor system and the safety device for deployingthe safety device when initiated by the anticipatory sensor system. Theanticipatory sensor system may include one or more receivers forreceiving waves or energy and a pattern recognition system for analyzingthe received waves or energy or data representative of the receivedwaves or energy to assess the probable severity of the impact. Thepattern recognition system is arranged to ascertain the identity of anobject from which the waves or energy have been emitted, reflected orgenerated, so that the assessment of the probable severity of the impactis at least partially based on the identification of the object. Thepattern recognition system includes a processor embodying a patternrecognition algorithm designed to provide an output of one of a numberof pre-determined identities of the object, e.g., a human or morespecifically a pedestrian or a cyclist or an animal.

The safety device can be an occupant protection apparatus arranged toprotect an occupant of the vehicle in the impact, such as one includingan inflatable airbag and an inflator for inflating the airbag.Alternatively, the airbag can deploy external to the vehicle at alocation between the object and an expected point of impact of theobject with the vehicle, e.g., to protect a pedestrian when impactingthe vehicle. The safety device may also be a net designed to capture anobject outside of the vehicle. Multiple safety devices may be deployedor initiated by the anticipatory sensor system.

The sensor system can include a transmitter for transmitting energy orwaves away from the vehicle and which are receivable by the wavereceiver(s) after reflection from the object, and which transmitter maybe arranged apart from the wave receivers. The transmitter may generatea visible or infrared laser beam, in which case, the wave receivers canbe a charge coupled or CMOS device which receive reflected light. Thesensor system can also include circuitry coupled to the wave receiversto process signals from the wave receivers representative of the wavesreceived by the wave receivers into an indication of the probableseverity of an impact between the object and the vehicle. Use of thesystem can be for side impacts, in which case, the wave receivers arearranged on a side of the vehicle to sense a pending side impact, or ata front of the vehicle to sense a pending frontal impact. Thetransmitter may transmit infrared waves, in particular those in aneye-safe range. The transmitter may be controlled by a controller orprocessor in the anticipatory sensor system such that the intensity oftransmitted infrared waves is determined at least in part by thedistance to the reflecting object, this distance may be obtained fromreceived waves or another distance measurement technique. Theanticipatory sensor system may also be arranged to apply range gating toselect a range of an object to be identified, and then analyze thereceived waves accordingly.

The safety device may be a net housed in a module along a side of thevehicle, e.g., along a front of the vehicle. In this case, only when thepattern recognition system identifies the object as a pedestrian doesthe anticipatory sensor system initiate the actuator to deploy the netto capture the pedestrian and hold it onto the vehicle after the impact.Thus, when the pattern recognition system identifies the object as anon-human animal, the anticipatory sensor system withholds initiation ofthe actuator.

The pattern recognition system may be designed to receive images fromthe wave receiver(s), if they are optical receivers, and analyze imagesof the object in order to ascertain the identity thereof. This analysismay involve image subtraction or analysis of multiple images obtainedduring variant imaging conditions, e.g., when the object is illuminatedby waves transmitted from the vehicle and in the absence of suchilluminating waves.

A method for protecting an occupant of a vehicle during an impactbetween the vehicle and an object in accordance with the inventionincludes detecting an impending impact between the vehicle and theobject prior to the impact, determining the probable severity of theimpending impact prior to the impact, and initiating deployment of asafety device outside of the vehicle between the vehicle and the objectin the event the impending impact is determined to be above a thresholdseverity. The determination of the probable severity of the impact mayentail receiving waves or energy from the object and ascertaining theidentity of the object based on the received waves or energy byinputting the received waves or energy or data representative of thereceived waves or energy into a pattern recognition algorithm whichascertains the identity of the object based thereon, the patternrecognition algorithm being designed to provide an output of one of anumber of pre-determined identities of the object.

The safety device may be housed in the front of the vehicle and may bean externally deployed airbag or net. The net may be positioned anddesigned to capture the object and hold it onto the vehicle after theimpact. However, the airbag or net may be controlled to deploy only whenthe object is a human such that when the object is a non-human animal,deployment of the airbag or net is not initiated. To this end, thepattern recognition algorithm may be designed to identify at least oneof a human, pedestrian, animal and a cyclist. Further, the airbag may bearranged to cover a portion of a vehicle hood when deployed and therebycontrol a trajectory of the object.

Detecting the impending impact may involve transmitting waves away fromthe vehicle, receiving the transmitted waves after the waves have beenreflected from the object and analyzing the received waves. Thetransmitted waves may be infrared waves, preferably in an eye-saferange. The intensity of the infrared waves may be determined based atleast in part on a distance to the reflecting object.

BRIEF DESCRIPTION OF THE DRAWINGS

The following drawings are illustrative of embodiments of the inventionand are not meant to limit the scope of the invention as encompassed bythe claims.

FIG. 1 is a perspective view of an automobile showing a preferredmounting location for the optical blind spot detection system inaccordance with the invention.

FIG. 2 is a perspective view of the vehicle of FIG. 1 shown operating ona highway.

FIG. 3A is a detailed view of an automobile mirror assembly showing thelocation of a LED array or a scanning laser and the CMOS camera.

FIG. 3B is a detailed view of a truck mirror assembly showing thelocation of a LED array or a scanning laser and the CMOS camera.

FIG. 4 is an overhead view of an alternate blind spot monitoring systemwhere the light source and camera are not collocated and some additionalcollocated corner mounted examples.

FIG. 5 is a view similar to FIG. 2 however showing the pattern of laserdiode illumination projected from a vehicle mirror preferredinstallation.

FIG. 6A is a top view of a large truck vehicle showing the coverage of aside blind spot area.

FIG. 6B is a side view of the large truck vehicle of FIG. 6A showing thecoverage of a side blind spot area.

FIG. 6C is a side view of the large truck vehicle of FIG. 6A showing thecoverage of a side blind spot area using multiple cameras along the sideof the vehicle.

FIG. 7A is a top view illustrating the coverage of the forward righthand side of a truck vehicle blind spot.

FIG. 7B is a side view illustrating the coverage of the forward righthand side of a truck vehicle blind spot.

FIG. 8A is a side view illustrating the application to a bus formonitoring the space in front of the bus.

FIG. 8B is a front view illustrating the application to a bus formonitoring the space in front of the bus.

FIG. 9 is a top view of a system applied to monitor the rear of thetruck trailer to protect for backup accidents.

FIG. 10 is a block diagram illustrating the blind spot detector,steering control, display and warning system.

FIG. 11 shows an icon display for the instrument panel and an alternateheads up display indicating the position of the host vehicle and thepositions of surrounding potentially threatening vehicles as seen fromabove.

FIG. 12 is an illustration similar to FIG. 11 showing the projection ofthe images onto a heads-up display.

FIG. 13A illustrates a view of the image as seen by a side rear viewcamera of FIG. 1.

FIG. 13B illustrates a view of the image of FIG. 13A after a stage ofimage processing.

FIG. 13C illustrates a view of the image as FIG. 13B after the vehiclehas been abstracted.

FIG. 14 illustrates a lane change problem in congested traffic.

FIG. 15 is a perspective view of a vehicle about to impact the side ofanother vehicle showing the location of the various parts of theanticipatory sensor system of this invention.

FIG. 15A is an enlarged view of a portion of FIG. 15.

FIG. 16 is a flowchart for the lane change problem.

FIG. 17 is an overhead view of a vehicle about to be impacted in thefront by an approaching vehicle showing a wave transmitter part of theanticipatory sensor system;

FIG. 18A is a plan front view of the front of a car showing theheadlights, radiator grill, bumper, fenders, windshield, roof and hood;

FIG. 18B is a plan front view of the front of a truck showing theheadlights, radiator grill, bumper, fenders, windshield, roof and hood;

FIG. 19 is an overhead view of a vehicle about to be impacted in theside by an approaching vehicle showing an infrared radiation emanatingfrom the front of the striking vehicle and an infrared receiver part ofthe anticipatory sensor system;

FIG. 20 is a side view with portions cutaway and removed of a dualinflator airbag system with two airbags with one airbag lying inside theother;

FIG. 21 is a perspective view of a seatbelt mechanism illustrating adevice to release a controlled amount of slack into seatbelt allowing anoccupant to be displaced;

FIG. 22 is a front view of an occupant being restrained by a seatbelthaving two anchorage points on the driver's right side where the one isreleased allowing the occupant to be laterally displaced during thecrash;

FIG. 22A is an expanded view of the release mechanism within the circledesignated 22A in FIG. 22;

FIG. 22B is a view of the apparatus of FIG. 22A within the circledesignated 22B and rotated 90 degrees showing the release mechanism;

FIG. 23 is a front view of an occupant being restrained by a seatbeltintegral with seat so that when seat moves during a crash with theoccupant, the belt also moves allowing the occupant to be laterallydisplaced during the crash;

FIG. 24A is a front view of an occupant being restrained by a seatbeltand a linear airbag module attached to the seat back to protect entireoccupant from his pelvis to his head;

FIG. 24B is a view of the system of FIG. 24A showing the airbag in theinflated condition;

FIG. 25A is a front view of an occupant being restrained by a seatbeltand where the seat is displaced toward vehicle center by the deployingairbag in conjunction with other apparatus;

FIG. 25B is a front view of an occupant being restrained by a seatbeltand where the seat is rotated about a vertical axis in conjunction withother apparatus;

FIG. 25C is a front view of an occupant being restrained by a seatbeltand where the seat is rotated about a longitudinal axis in conjunctionwith other apparatus;

FIG. 26A is a perspective view with portions cutaway and removed of avehicle about to impact the side of another vehicle showing an airbagstored within the side door of the target vehicle prior to beingreleased to cushion the impact of the two vehicles;

FIG. 26B is a view of the apparatus of FIG. 26A after the airbag hasdeployed;

FIG. 27 is a schematic drawing of a variable inflation inflator systemin accordance with the invention using two inflators;

FIG. 28 is a schematic drawing of a variable inflation inflator systemin accordance with the invention using a single inflator and a variableoutflow port or vent;

FIG. 29A shows a situation where a vehicle equipped with an externallydeployable airbag for frontal impact protection is about to impactanother vehicle;

FIG. 29B shows the condition of the vehicles of FIG. 29A at impact;

FIG. 30A shows a situation where a vehicle equipped with an externallydeployable airbag for rear impact protection is about to be impacted byanother vehicle;

FIG. 30B shows the condition of the vehicles of FIG. 30A at impact;

FIG. 31A shows a situation where a vehicle equipped with an externallydeployable airbag for frontal impact protection is about to impact apedestrian;

FIG. 31B shows the condition of the vehicle and pedestrian of FIG. 31Aat impact;

FIGS. 32A, 32B and 32C show one use of positioning airbags in accordancewith the invention wherein a passenger is shown leaning against a doorin FIG. 32A, a positioning airbag deploys to move the passenger awayfrom the door as shown in FIG. 32B and a side curtain airbag is deployedwhen the passenger has been moved away from the door as shown in FIG.32C;

FIGS. 33A, 33B, 33C and 33D show the manner in which an occupant can bepositioned improperly for deployment of a side curtain airbag during arollover leading to severe injury;

FIGS. 34A, 34B, 34C and 34D show the manner in which an occupant isre-positioned for deployment of a side curtain airbag during a rolloverthereby preventing severe injury;

FIG. 35 is a flow chart showing the manner in positioning airbags areused in accordance with the invention;

FIG. 36 is a schematic of the apparatus for deploying multiple airbagsin accordance with the invention;

FIG. 37 is a block diagram of a known agricultural product distributionmachine;

FIG. 38 is a block diagram of a control unit, in accordance with anembodiment of the primer system;

FIG. 39 represents a basic air delivery system;

FIG. 40 represents a metering mechanism of the air delivery system inFIG. 39;

FIG. 41 is a block diagram of a primer system in accordance with anembodiment of the present invention, as it applies to seeders andplanters;

FIG. 42 represents a basic sprayer system;

FIG. 43 shows a rearward pointing installation arrangement for a vehiclespeed detection system on a bulldozer according to an embodiment of theinvention;

FIGS. 44A and 44B show the effects of slope and pitch on the groundspeed sensor of this invention;

FIG. 45 shows a forward pointing installation arrangement for a vehiclespeed detection system on a tractor according to an embodiment of theinvention;

FIG. 46 is a block diagram for an ultrasonic distance and speeddetector;

FIG. 47 is a schematic of a circuit for use in determining the distanceto object; and

FIGS. 48 and 49 show a situation where a vehicle equipped with anexternally deployable airbag for frontal impact protection and netcatches a pedestrian impacting the vehicle.

DETAILED DESCRIPTION OF THE INVENTION

1. Exterior Monitoring

1.1 General

Referring now to the drawings wherein the same reference numerals referto like elements, a perspective semi-transparent view of an automobileis shown generally as 1 in FIG. 1. A driver 2 of the automobile sits ona front seat 4. Five transmitter and/or receiver assemblies 5, 6, 7, 8and 9 are positioned at various places with views of the environmentsurrounding the vehicle. Any of these assemblies can have multipletransmitters and/or receivers. A processor such as control circuitry 12is connected to the transmitter/receiver assemblies 5-9 by appropriatewires, not shown, or wirelessly and controls the transmission of wavesor energy from the transmitter portion of the assemblies 5-9 andcaptures the return signals received by the receiver portion of theassemblies 5-9. Control circuitry 12 usually contains one or more analogto digital converters (ADCs) or frame grabbers, a microprocessorcontaining sufficient memory and appropriate software including patternrecognition algorithms, and other appropriate drivers, signalconditioners, signal generators, etc. Usually, in any givenimplementation, only two to four of the transmitter/receiver assemblieswould be used. Some or all of the processor functions can take place inthe assemblies 5-9.

These optical, for example, transmitter/receiver assemblies 5-9, alsoreferred to herein as transducer assemblies or transceivers, arecomprised of an optical transmitter or light emitting component, whichmay be an infrared LED, a high power laser diode forming a spotlight, alaser with a diverging lens, a floodlight or a scanning laser assembly,any of which can be pulsed or modulated, and a receiver such as a CCD orCMOS array or pin diode or equivalent photo detector. If modulation isused, it can be frequency, amplitude or pulse-modulated and themodulation scheme can be sine wave or code and if code, the code can berandom or pseudorandom. Preferably, a transmitter/receiver assemblycomprises an active pixel CMOS array or an HDRC array as discussedbelow. The transducer assemblies map the location of the objects andfeatures thereof, in a two and/or three-dimensional image as will alsobe described below. In a preferred design, range gating is used tolocate objects in the field of view and aid in separating an object ofinterest from other objects and the background. Range gating is alsoused to aid in low visibility situations such as in fog, smoke, rain andsnow.

In one preferred embodiment, assemblies 5-9 are high-powered laser flashsources that emit very short IR pulses, which can be in the eye-safepart of the spectrum, to illuminate the area surrounding the vehicle andrange gating techniques can be used to determine the distances toreflecting objects. The laser on/off duty cycle is kept small and thetiming of the flash can be determined by the vehicle location (such asits GPS coordinates) such that the emissions from one vehicle are notconfused by or with emissions from other nearby vehicles. Such systemscan also be used with auto focus lenses or other optical devices tocontrol the beam pattern.

The foregoing examples of possible wave/energy/light emitting componentsand light/wave/energy receiver components are not intended to limit theinvention and it should be understood by those skilled in the art thatother transmitter and receiver components and combinations can be usedin accordance with the invention without deviating from the scope andspirit thereof.

In a preferred embodiment, four transducer assemblies 5-8 are positionedaround the exterior of the vehicle in the spaces to be monitored, eachcomprising one or more LEDs or laser diodes and a CMOS array with alight valve and an appropriate lens. Although illustrated together, theilluminating source will frequently not be co-located with the receivingarray particularly when triangulation distance measurement is used, asdescribed below. The LED, laser or other appropriate source ofillumination can emit a controlled angle diverging beam of infraredradiation that illuminates a particular space and illuminates an objectat a particular point that depends on the location of the objectrelative to the vehicle and the direction of the LED or laser beam, forexample. In some applications, the beam does not diverge and in others,the beam converges.

The image from each array is used to capture two or three dimensions ofobject position information, thus, the array of assembly 5, which can belocated approximately behind the driver's door on the B-pillar providesboth vertical and transverse information on the location of an object inthe vicinity of the vehicle. A similar view from a location on thepassenger side is obtained from the array of assembly 6. The mountinglocations of the assemblies 5, 6 shown in FIG. 1 are exemplary and arenot intended to limit the possible positions for placement of theassemblies. Other positions for installation of the assemblies on thesides of the vehicle are contemplated. For example, the assemblies 5, 6could be placed on the side of the vehicle alongside the passengercompartment, engine compartment or trunk compartment.

If the receiving array of assembly 5 contains a matrix of 100 by 100pixels, then 10,000 pixels or data elements of information will becreated each time the system interrogates the space on the driver sideof the vehicle, for example. Interrogation of the space on the driverside of the vehicle would entail commanding the assembly 5 to transmitoptical waves or energy into the environment surrounding the vehicle bymeans of the transmitter component of the assembly 5 and receiving anyreflected optical waves or energy by the receiver component of theassembly 5.

There are many pixels of each image that can be eliminated because theydo not contain any useful information. This typically includes thecorner pixels and other areas where an object cannot be located. Thispixel pruning can typically reduce the number of pixels by up to 20percent resulting in approximately 8,000 remaining pixels, for example.The output from each array is then preferably preprocessed to extractthe salient features and fed to an artificial neural network, or otherpattern recognition system, to identify the object or ascertain theidentity of the object. The preprocessing can include edge detection anda variety of filters such as described in U.S. patent application Ser.No. 11/025,501 filed Jan. 3, 2005. Range gating also can be used toeliminate reflections from unwanted objects and to reduce the effects offog, smoke, rain and snow, for example, as described below.

The preprocessing step frequently makes use of distance or relativemotion information to separate one object from another and from otherparts of the captured scene. Once this operation is completed for all ofthe object images, the identification of the objects in the spaceproximate to the driver side of the vehicle has been determined.

The feature extraction frequently involves identifying the edges ofobjects in the field of view and the angular orientation of the foundedges. The locations of the edges and their orientation can then beinput into appropriate recognition software. Other feature extractiontechniques are also applicable.

A pattern recognition technique such as a trained neural network can beused to determine which of the trained occupancies most closelycorresponds to the measured data. The output of the neural network canbe an index of the setup that was used during training that most closelymatches the current measured state. This index can be used to locatestored information from the matched trained occupancy. Information thathas been stored for the trained occupancy typically includes the locusof the centers of different objects and an appropriate icon. For thecase of FIG. 1, it is also known from one of the techniques to bedescribed below where the object is located relative the vehicle.

There are many mathematical techniques that can be applied to simplifythe above process. One technique used in military pattern recognition,for example, uses the Fourier transform of particular areas in an imageto match by correlation with known Fourier transforms of known images.In this manner, the identification and location can be determinedsimultaneously. There is even a technique used for target identificationwhereby the Fourier transforms are compared optically. Other techniquesutilize thresholding to limit the pixels that will be analyzed by any ofthese processes. Still other techniques search for particular featuresand extract those features and concentrate merely on the location ofcertain of these features.

A particularly useful technique, as mentioned above, calculates thelocation of the edges of the object in the blind spot and uses theseextracted edges as the features that are fed to the neural network.(See, for example, the Kage et al. artificial retina paper referencedabove which, together with the references cited therein, is incorporatedherein by reference.)

1.2 Blind Spots

1.2.1 General

In the discussion below, blind spots are used as an example ofmonitoring the space around the vehicle. Although the phrase “blindspot” is used, it is the intention of the inventor that this is merelyan example of monitoring the exterior of the vehicle and thus “blindspot” will be used as a surrogate for monitoring of any area surroundingthe vehicle from the vehicle.

One principle used in this embodiment of the invention is to use imagesof different views of an object in the blind spot to correlate withknown images that were used to train a pattern recognition system suchas a neural network for blind spot occupancy. Then, carefully measuredpositions of the known images are used to locate particular parts of theobject such as the windshield, tires, radiator grill, headlights, etc.

One important point concerns the location and number of opticalassemblies. For an automobile, one assembly is generally placed on eachside of the vehicle such as shown by assemblies 5, 6. In someembodiments, a third assembly 7 can be placed to view the blind spotbehind the vehicle and a fourth assembly 7 can be placed to view infront of the vehicle for automatic cruise control, for example. Use ofan optical imaging assembly as a cruise control device permits onedevice to perform the functions of providing information for bothmaintaining an appropriate distance behind another vehicle, as in anautomatic cruise control system, and collision avoidance andanticipatory sensing functions.

The methods by which the principles herein described can be carried outare numerous. In one preferred embodiment, four assemblies are placed onthe four sides of the vehicle with each assembly having a field of viewof 180 degrees. A cylindrical lens or equivalent is used to capture the180 degree field of view. Sections of the view can be selectivelyanalyzed and similarly sections can be selectively illuminated as thesituation requires. For illumination, either the illuminating IR laserdiode can be selectively directed to illuminate smaller angular sectionsof the view, a more powerful laser diode can be used to illuminate theentire 180 degrees, multiple laser diodes can be used or otheralternatives. The same can be applied to CMOS imagers. Either the imagercan be caused to change its field of view, or a specially designedimager that images the full 180 degrees or multiple imagers can be used.As mentioned above, the distance to objects within the field of view canbe determined through a variety of methods including range gating. Theillumination timing of the laser diode(s) can be a function of thevehicle location, such as the DGPS coordinates, so as not to interferewith systems on other vehicles. The IR illumination source can be a highpowered (1-100 transmitted watts) device operating in the eye-safe partof the IR spectrum (>1.4 microns)

1.2.2 Range Gating

An alternate approach is to make a three-dimensional map of the objectin the blind spot based on the optical energy or waves received by thereceiver components of the assemblies 5-9 and to precisely locate thesefeatures using neural networks, fuzzy logic or other pattern recognitiontechniques. One method of obtaining a three-dimensional map is toutilize a laser radar (lidar) system where the laser is operated in apulse mode and the distance from the object being illuminated isdetermined using range-gating in a manner similar to that described invarious patents on micropower impulse radar to McEwan. (See, forexample, U.S. Pat. No. 5,457,394 and U.S. Pat. No. 5,521,600). Rangegating is also disclosed in U.S. patent application Ser. No. 11/034,325filed Jan. 12, 2005 assigned to ITI. Alternatively, the laser can bemodulated and the phase of the reflected and the transmitted light canbe compared to determine the distance to the object or random orpseudorandom modulation can be used with correlation techniques.

1.2.3 Scanning

The scanning portion of the laser radar device can be accomplished usingrotating mirrors or prisms, galvanometer mirrors, MEMS mirrors orpreferably, a solid state system, for example an acousto-optical deviceutilizing TeO₂ as an optical diffraction crystal with lithium niobatecrystals driven by ultrasound (although other solid state systems notnecessarily using TeO₂ and lithium niobate crystals could also be used).An alternate method is to use a micromachined mirror, which is supportedat its center or edge and caused to deflect by miniature coils orelectrostatically. Such a device has been used to providetwo-dimensional scanning to a laser. This has the advantage over theTeO₂—lithium niobate technology in that it is inherently smaller andlower cost and provides two-dimensional scanning capability in one smalldevice. The maximum angular deflection that can be achieved with thisprocess is on the order of about 10 degrees. A diverging lens or mirror,or similar technique, can be used to achieve a greater angular scan ifnecessary. An alternate preferred approach is to use passive opticalimages with superimposed infrared dots created by an array of infraredlaser diodes in a manner similar to that described in U.S. Pat. No.6,038,496. Other forms of structured light can of course be used. Analternate approach uses a digital light processor (DLP) from TexasInstruments which can contain on the order of 1 million mirrors.

In the DLP implementation, the hardware for the scanner includes a DLPaddressable mirror array wherein each mirror can rotate plus or minus 12degrees and wherein the array includes up to one million or moremirrors. Each mirror can be moved at a rate of 40 KHz. Additionalhardware includes cylindrical lenses, one or more CMOS cameras, a ten toone hundred watt laser diode having a wavelength of approximately 1.5microns, means for pulsing the laser and one or more light valves orspatial light monitors located in front of the cameras.

Information on the DLP can be found at dlp.com.

The DLP is capable of reflecting the light from the laser diode, orother illumination source, in one million directions assuming that theDLP has one million mirrors. These beams can pass through a properlydesigned cylindrical lens which can have the effect of multiplyingmotions of the beams in the horizontal direction but not in the verticaldirection, for example. The beam that passes through the cylindricallens can have its angle increased, for example, by a factor of 4 or 5.In so doing, the shape of the beam will become elliptical with a ratioof horizontal to vertical dimensions of 4 or 5 to 1. In the aboveexample, 1 to 5 cameras can be selected depending on the portion of thescene and resolution desired. These cameras can also each have acylindrical lens which can have the effect of increasing the horizontalfield of view relative to the vertical. This compression can be reversedduring signal processing to create an undistorted partially panoramicimage.

Under normal operation when there are no objects of interest in theviewing area, the one million pixels would be spread out as a pattern ofbeams projecting down the highway and to either side. The camerasmonitoring the deflected beams would operate in a low resolution mode.Whenever an object was discovered in the view that could potentially bethreatening, the DLP will shift some of its beams to provide a morecomplete illumination of the object and the cameras will shift into ahigher resolution mode. If multiple cameras are used, then only thecamera that images the object would have to be in the high resolutionmode. Based on when the object came into the scene and which beams firstilluminated the object, an approximate range to the object can usuallybe determined. Range gating can then be used to segment the object foridentification and also to determine more precisely where the object isin its velocity. The object can be further monitored until it is nolonger threatening. Depending on the computational capacity of thesystem, and assuming that ten thousand pixels are sufficient toilluminate one object, the million pixel DLP could track up to onehundred objects in its field of view simultaneously.

Power problems are eliminated by using eye safe lasers. Eye safe lasershave been used by the military at up to one million watts. (See VerleAebi and Peter Vallianos “Laser-illuminated viewing provides long-rangedetail”, Laser Focus World, September 2000).

The range gating system should be coordinated with the laser pulsingsystem. The laser pulsing system can be based on the geographicallocation of the vehicle in such a manner that vehicles in the vicinityof each other would not confuse each other's system. If ten or even onehundred slots are managed in this matter there should be little or nointerference between vehicles' systems.

The same system can be used to monitor the highway for fiduciary objectsthat have been previously identified during the mapping process. Theprecise distance to these fiduciary objects can be measured and in facttracked as the vehicle passes such an object. This becomes a far simplermethod of getting precise positioning independent of GPS than theearlier signature matching concepts.

Actually, the system or a similar system can be used inside the vehicleallowing the occupant monitor to concentrate only on the occupant andnot on the surrounding vehicle interior and especially windows. Whenused inside the vehicle, it can also be used for anti-trap functionssuch as for windows and doors. For inside the vehicle, a cylindricallens would be replaced by another convenient lens.

1.2.4 Structured Light

An alternate method of obtaining three-dimensional information from ascanning laser system is to use multiple arrays of IR dots, or otherform of structured light, to replace the single arrays used in FIG. 1.In the case, the arrays are displaced from each other and, throughtriangulation, the location of the reflection from the illumination by alaser beam of a point on the object can be determined by triangulationand/or correlation in a manner that is understood by those skilled inthe art. Of course, the arrays can be timed differently if desired tofurther enable the correlation of the reflection with the array source.

1.3 Optical Methods

1.3.1 Single 360 Degree Optical System

An alternate configuration is shown at assembly 9 which is a lensarrangement which provides a view of 360 degrees by approximately 20degrees. Although this camera does not provide as complete a view ofobjects in the various blind spots, it is possible using a single deviceto observe areas on both sides as well as the front and back of thevehicle. The same lens is used for receiving the images and forprojecting a rotating scanning laser beam, for example, thatapproximately bisects the 20-degree angle. This rotating laser beam ismodulated thereby permitting the distance to the reflected laser lightto be determined. A rotating mirror or prism, that also serves todeflect the laser beam, captures the returned laser light. This mirroror prism is positioned so that it is above the portion of the lens usedfor receiving the images such that laser system does not interfere withthe imaging system. Again, in a preferred implementation, the laser is ahigh powered spotlight in the eye-safe IR portion of the electromagneticspectrum.

Special lenses can be used to collect the light from the sphericalsegmented lens and project the combined image onto a CMOS imager. Insome cases, software is provided to remove known distortions for imageanalysis or, in other cases, this is not necessary as the patternrecognition system has been trained on the combined received image, or asegmented version thereof which divides the image into, for example,four segments representing front, right, rear, and left quadrants.

1.3.2 Laser Flash Systems—General

One preferred method of monitoring the space surrounding a vehicle is touse a laser flash system as discussed in U.S. Pat. No. 7,202,776. Inthis embodiment, a laser beam is projected into the space of interest.It can be a broad spotlight beam that illuminates a substantial area orit can be a narrow scanning beam. The beam can be made to diverge,converge or remain of approximately constant diameter. It is generallyin the near infrared portion of the electromagnetic spectrum. The beamcan be modulated in any manner such as by a sine or other periodic waveor by pulse modulation. The modulation can be varied as to amplitude orphase or by random, pseudorandom or code modulation or any combinationthereof. Of particular interest is that the laser beam, whethermodulated or not, can be sent in bursts permitting the distance to anobject to be determined or the reflections from in front of or beyond anobject or particular range to be range gated thereby permitting an imageto be obtained, for example, of an object at a particular distance fromthe vehicle. This range gating can be accomplished through a variety ofspatial light monitors including Pockel or Kerr cells, garnet crystals,liquid crystals or the equivalent as discussed in the '776 patent.

Use can be made also of the fact that infrared is more easily absorbedby atmospheric moisture such as fog and snow than reflected so that thepenetration of the IR laser can be increased when such atmosphericmoisture is present by increasing the emitted power and by reducing thedivergence of the beam. In some cases, the beam can be made to convergethrough appropriate lenses to compensate for the absorption andscattering of the laser light. In one embodiment, a narrow beam IR laseris used to monitor the atmospheric absorption and thus to control thelight emitted by the main illuminating laser system. In this case, themonitoring laser sends out a beam of IR and samples the return at eachdistance from the source using range gating. A map can then bedetermined of the absorption plus scattering as a function of distancefrom the source and thus the maximum IR radiation that can be emittedand the beam angle from the main illuminating laser system determined sothat at no distance from the source does the intensity exceed eye safetylimits. Note that research is underway to permit control of thefrequency emitted by the laser system thus permitting a matching of thefrequency as well as the intensity and beam angle to the atmosphericconditions. Once again, preferred IR wavelengths used are in theeye-safe portion of the spectrum. By carefully selecting the exactfrequency emitted to coincide with solar minimums, the chance of thereflected light being overpowered by emissions from the sun isminimized.

When the atmospheric adjustment system described above is coupled withrange gating, distances to, and images of, objects of interest can beobtained for ranges exceeding 500 meters in low visibility conditions.Thus, the system of this invention can replace radar for collisionavoidance, automatic cruise control, blind spot monitoring and all otheraround vehicle monitoring applications. The laser atmospheric monitoringsystem can also be used to determine visibility conditions which canthen be used to set the maximum speed the vehicle is permitted totravel. This visibility information as a function of distance can alsobe communicated to other vehicles either by direct communication orindirectly through infrastructure-based transceivers or through theInternet, providing the vehicle has an Internet connection, where it canthen be transmitted to vehicles in the vicinity as a map update, forexample.

As disclosed elsewhere herein and in the patents and patent applicationsof the current assignee, the modulation scheme can also be used todetermine the distance to an object either alone on in conjunction withrange gating. Code, random or pseudorandom modulation is particularlyuseful for this function however even simple sine wave modulation canpermit the more accurate location of an object within a range gatedsignal.

When the distance to an object is the main function of the laser system,additional range can be obtained if the receiving beam is focused onto asimple pin or avalanche diode receiver. This system is enhanced evenmore if the IR is polarized to permit the partial elimination of noisefrom the atmosphere. Range gating can be used along with variousmodulation schemes as desired by the designer. Naturally both systemscan be used where an imager is used for identification and a parallelpin diode for ranging.

1.3.3 Laser Flash Systems—Timing

A potential problem can arise when several vehicles have the same or asimilar system and the systems then can be multiplexed in a variety ofways to prevent interference. These include time, code and frequencymethods that can be managed based to the sensing of output from theother vehicles and/or by a method based or the GPS clock and/or theaccurate coordinates of the vehicle perhaps coupled with its velocityvector. Any of these can additionally employ polarization methods. Someof these combinations will now be discussed but this inventioncontemplates all combinations of the above mentioned interferenceavoidance methods and others which are used to control the emission ofIR light beams from laser illumination systems that are used to monitorthe space around the vehicle.

If broad beam illumination is used and simple imaging techniques areemployed, then the illumination from other vehicles may not interferewith the system especially in high dynamic range cameras are used thatminimize the effects of one vehicle's system blinding another. Suchcameras can adjust the sensitivity to received illumination on apixel-by-pixel basis (see, e.g., products produced by IMS chips ofStuttgart Germany and other active pixel cameras available from othermanufacturers). When it is deemed desirable to eliminate theinterference from another vehicle, one simple method is similar to theEthernet protocol where when one vehicle senses the transmission fromanother vehicle it backs off for a random delay and then retries. Sincethe duty cycle for such a system can be very small, less that 1% forexample, such a system can permit hundreds of vehicles to operatesimultaneously with each vehicle obtaining its desired information. Thismethod is an inefficient use of resources, however, and other moresophisticated methods may be required. If each vehicle modulates itstransmissions with a pseudorandom or other code, then the reflectionsfrom a particular vehicle's transmissions can be filtered out usingcorrelation techniques. This can require a significant computationalload if all of the pixels of an image need to be individually analyzed.Naturally, each vehicle can operate at a slightly different frequencyand if variable frequency laser systems are available, then this becomesa viable alternative, provided there is an algorithm that instructs eachvehicle as to the frequency that it should use. A similar commentapplies to different polarization angles for vehicles using the samefrequency. Such an algorithm can be based on the location and/orvelocity vector of the vehicle as discussed below.

It is anticipated that the systems described here may be a part of amore extensive system that is designed to substantially reduce vehicleaccidents and congestion such as described in the '325 applicationreferenced above. For this case, each equipped vehicle will know itsexact location and velocity vector. It will also know the time veryaccurately based on the atomic clocks resident in the GPS satellites.Each vehicle can then operate its laser monitoring system in aparticular direction during an allotted time period where that timeperiod is determined based on the vehicle's location and velocity vectorin such a manner that no two vehicles within a range of each other, forexample, will use the same time slice in a direction of transmissionthat it would interfere with any other vehicle within that range. Thisrange can be fixed, a function of the vehicle location or a function oftraffic density or other consideration such as geographicalconsiderations which can be found on vehicle resident maps.

An important application of the laser illumination system disclosedherein is to provide an image of objects that may pose a threat to thesystem-resident vehicle. In this case, the image can be analyzed using apattern recognition algorithm and the results either displayed as anicon in the field of view of the driver or pilot using a heads-updisplay or the results of the analysis can be used to cause an audio,haptic or visual warning to be issued to the vehicle operator. In somecases, the results of the analysis can be used to control the vehiclethrough its braking or steering system or by some other appropriatemeans. In general, this system is not intended as a general roadillumination system. In some cases, the vehicle will have a residentaccurate map of the roadway which will permit the vehicle laserillumination system to know where the road ahead is and thus where topoint the illumination system so that it covers the lane on which thevehicle is driving and, through range gating, even those parts of thelane that are on curves and hills. For more general illumination use ofa similar system, see U.S. patent application Pub. Nos. 20020191388,20030193980, 20030193981 and 20030198271.

Although the systems discussed above use active IR illumination, passiveIR can also be used to complement the active system. A passive IR systemcan frequently spot animals or vehicles in locations where they may bemissed by the active system which can then be used to direct theattention to the active system. The active system may be concentratingon the road ahead and thus miss a deer or pedestrian that is about toenter the path of the vehicle but has not yet done so.

The discussion above on active IR has mainly been concerned with thenear IR range which has wavelengths below 1400 nanometers and where eyesafety is still a problem. Developments in the SWIR range (particularlyin the range of 1400 nm to 1700 nm) using indium gallium arsenide(InGaAs) for an imager permit much higher power transmissions as theyare below the eye safety zone (see, e.g., Martin H. Ettenberg “A LittleNight Vision”, Solutions for the Electronic Imaging Professional, March,2005, a Cygnus Publication available at sensorsinc.com.

1.3.4 Pattern Recognition

As mentioned elsewhere herein, a key component of many of the inventionsdisclosed herein is the use of pattern recognition technologies toidentify objects and makings on the roadway and/or in its vicinity. Thepattern recognition of choice is based on neural networks in all of itsvarious forms including associative memory systems, modular neuralnetworks, cellular neural networks, support vector machines andcombination neural networks. Of course, pattern recognition alsocontains within its definition other more crude methods such as patternmatching and the use of templates. Objects that can be recognized usingthis technology include animals such as deer, lane marking lines, othervehicles, pedestrians, stop signs and stop lights including the color ofthe light (red, yellow, green), speed limit and other roadside signs,ice and snow on the roadway, character of the shoulder and area beyondthe shoulder, trees, motorcycles, bicycles, light posts, and all otherobjects that are commonly found on or along side of a road.

All types of collision avoidance systems including lane departure orroad departure warning systems can use the pattern recognitiontechnologies disclosed herein.

1.3.5 Combination of Vision, Accurate Maps and IMU for ExternalMonitoring

Monitoring the space surrounding a vehicle can be used to get a measureof the motion of the vehicle and of the time to contact a vehicle asdescribed in U.S. Pat. No. 6,704,621, WO0139018 and U.S. patentapplication Pub. No. 20030040864. However, such methods are at bestrelatively inaccurate and computationally expensive. InertialMeasurement Units (IMUs) based on MEMS technology are being rapidlydeveloped and improving in accuracy. When corrected using DGPS oranother infrastructure-based location system, these devices can be usedon all vehicles where they will accurately monitor the motion of thevehicle and serve as chassis control, crash and rollover sensing, andnavigation systems. An IMU, however, can aid the exterior monitoringsystem by eliminating vibrations in the camera mounting and, inconjunction with accurate maps, aid in knowing where the monitoringsystem should be monitoring. The system will know when to look for stoplights, when there is an intersection approaching and where there is anincreased possibility for deer to be present, for example. Knowing whereto monitor will substantially reduce the computational load on thevision system resulting in system cost reduction.

A good discussion of the combined use of inertial sensors and camerascan be found in Michael Brownell, “Fusing Inertial Sensors withStereoscopic Cameras Advances 3-D Vision Systems” SPIE OEMagazine, March2004. This article talks about using the inertial devices to provide anaccurate ground plane from which the vision system can get itsreference. This can be useful in vehicle applications providing theinertial system, which would most likely be in the form of an IMU, has amethod of error correction such as provided by DGPS, or another positiondetermination system, such as described in the '325 applicationreferenced above. It might be possible to reverse the process and usethe vision system to correct the IMU but this may not have the requiredaccuracy.

1.3.6 Stereo Vision

Stereo vision has been suggested as a method for determining thelocation of an object relative to the host vehicle (see, e.g., U.S.patent application Pub. No. 20040252863). This approach doubles thenumber of cameras and perhaps illumination systems as well as increasesthe computational complexity and thus the cost of the system. There arealso problems with objects approaching directly in the direction of acamera pair and getting the required velocity vector accuracy forcollision avoidance. For these and other reasons, a laser radar approachdisclosed herein is preferred. The need to determine the position ofobjects relative to a vehicle will of course decrease as more and morevehicles are equipped with vehicle-to-vehicle communication and canbroadcast their positions and velocities to other vehicles. When allvehicles have such capability, the only remaining obstacles are animals,pedestrians, objects fallen from trucks, rocks from rock slides, fallentrees etc. These objects are unlikely to have an appreciable velocityand thus their position is all that is usually required. The position ofsuch objects can be more easily determined from the techniques hereindescribed.

The techniques of U.S. Pat. No. 6,704,621, WO0139018 and U.S. patentapplication Pub. No. 20030040864 are also a preferable approach to thecomplexity introduced by stereo cameras. Also, a structured lightapproach as described for interior vehicle monitoring in U.S. Pat. No.7,164,117 is a simpler and less expensive method of obtaining a 3D depthmap of the field of view than resorting to stereo cameras. Finally, asimilar approach is based on the knowledge of various geometric featuresof the road or nearby structures which are known from a vehicle residentmap database and using that knowledge to determine the location and/orsize of an object. For example, if it is known that the width of theroad lane on which the vehicle is traveling is 10 feet and an objectdown the road has a height that is equal to 50% of the road width atthat point, then the height of the object can be estimated to be 5 feet.Also, if the camera parameters are known the location of the objectrelative to the vehicle can be determined by conventional triangulationmethods known to those skilled in the art.

1.3.7 Miscellaneous

When monitoring the exterior space around a vehicle, optical methodshave been discussed. Optical infrared frequencies are absorbed more thanlower frequencies such as terahertz and millimeter wave radar. Radar, onthe other hand, has much lower resolution than optics but can moreeasily determine the distance to objects especially in the presence offog, rain and snow. One possibility is to fuse the two technologies sothat they cover the same area of exterior space and provide a compositeimage that combines the advantages of both systems. The optical systemwould provide the imaging capabilities when there is appropriatevisibility and the radar when there is not. Even in good visibilitysituations, the radar can provide the distance to objects in the fieldof view and the optics can aid in the interpretation of the radar image.The direction of the optical beam can be controlled through the rotationof a mirror, for example, and the direction of the radar beam throughsolid state steerable antenna technology as in a phased array antennasystem.

Various types of coatings can be applied to the lenses of the opticalsystems disclosed herein. Examples of some suitable coatings includingphotocatalytic and hydrophilic coatings as disclosed in U.S. Pat. No.6,193,378.

1.4 Combined Optical and Acoustic Methods

An ultrasonic system is the least expensive and potentially providesless information than the laser or radar systems due to the delaysresulting from the speed of sound and due to the wave length which isconsiderably longer than the laser systems. The wavelength limits thedetail that can be seen by the system. In spite of these limitations, asshown in Breed et al. (U.S. Pat. No. 5,829,782), ultrasonics can providesufficient timely information to permit the position and velocity of anapproaching object to be accurately known and, when used with anappropriate pattern recognition system, it is capable of positivelydetermining the class of the approaching object. One such patternrecognition system uses neural networks and is similar to that describedin the papers by Gorman et al. and in the rear facing child seatrecognition system referenced and described in the Breed et al. patentreferenced above.

The particular locations of the optical assemblies are selected toprovide accurate information as to the locations of objects in the blindspots. This is based on an understanding of what information can be bestobtained from a visual image. There is a natural tendency on the part ofhumans to try to gauge distance from the optical sensors directly. Thistypically involves focusing systems, stereographic systems, multiplearrays and triangulation, time-of-flight measurement, phase comparison,etc. What is not intuitive to humans is to not try to obtain thisdistance directly from apparatus or techniques associated with themounting location but instead to get it from another location. Whereasultrasound is quite good for measuring distances from the transducer(the z-axis), optical systems are better at measuring distances in thevertical and lateral directions (the x and y-axes).

For monitoring the interior of the vehicle, such as described in U.S.Pat. No. 6,324,453 this can more easily done indirectly by anothertransducer. That is, the z-axis to one transducer is the x-axis toanother. For external monitoring, a preferred approach, as describedbelow, is to use an array of LEDs or a scanning laser and locate theposition of the object in blind spot by triangulation, time-of-flight,range-gating or phase measurement although sometimes appropriatelylocated cameras in concert can provide three-dimensional informationdirectly (such as by stereo cameras).

Systems based on ultrasonics and neural networks, and optics and opticalcorrelation have been very successful in analyzing the seated state ofboth the passenger and driver seats in the interior of automobiles. Suchsystems are now in production for preventing airbag deployment when arear facing child seat or an out-of-position occupant is present. Theultrasonic systems, however, suffer from certain natural limitationsthat prevent the system accuracy from getting better than about 99percent. These limitations relate to the fact that the wavelength ofultrasound is typically between 3 mm and 8 mm. As a result, unexpectedresults occur which are due partially to the interference of reflectionsfrom different surfaces.

Additionally, commercially available ultrasonic transducers are tuneddevices that require several cycles before they transmit significantenergy and similarly require several cycles before they effectivelyreceive the reflected signals. This requirement has the effect ofsmearing the resolution of the ultrasound to the point that, forexample, using a conventional 40 kHz transducer, the resolution of thesystem is approximately three inches, although this has been improved.These limitations are also present in the use of ultrasound for exteriorvehicle monitoring.

In contrast, the wavelength of the portion of the infrared spectrum thatis contemplated for one preferred use in the invention is less than fivemicrons and no significant interferences occur. As a result, resolutionof the optical system is determined by the pixel spacing in the CCD orCMOS arrays or the speed of the pin or avalanche diode and scanner whenused. For this application, typical arrays have been selected to beapproximately 100 pixels by 100 pixels and therefore, the space beingimaged can be broken up into pieces that are significantly less than afew inches in size. If greater resolution is required, arrays havinglarger numbers of pixels are readily available.

Another advantage of optical systems is that special lenses can be usedto magnify those areas where the information is most critical andoperate at reduced resolution where this is not the case. For example,the area closest to the center of the blind spot can be magnified andthose areas that fall out of the blind spot, but are still beingmonitored, can be reduced. This is not possible with ultrasonic or radarsystems where it is even very difficult to get an image of sufficientresolution to permit an identification of the object to be accomplished.

Additional problems of ultrasonic systems arise from the slow speed ofsound and diffraction caused by variations in air density. The slowsound speed limits the rate at which data can be collected and thuseliminates the possibility of tracking the motion of an object moving athigh speed relative to the vehicle.

1.5 Discussion of the External Monitoring Problem and Solutions

In the embodiment shown in FIG. 1, transmitter/receiver assemblies 5-9may be designed to emit infrared waves that reflect off of objects inthe blind spot, for example, and return thereto. Periodically, theassemblies 5-9, as commanded by control circuit 12, transmits a pulse ofinfrared waves and the reflected signal is detected by a differentassembly. Alternately, a continuous scanning arrangement can be used.The transmitters can either transmit simultaneously or sequentially. Anassociated electronic circuit and algorithm in control circuit 12processes the returned signals as discussed above and determines theidentity and location of the object in the blind spot. This informationis then sent to a warning system that alerts the driver to the presenceof the object as described below. Although a driver side system has beenillustrated, a similar system can also be present on the passenger sideand can be applied to the front and rear of the vehicle.

The accuracy of the optical sensor is dependent upon the accuracy of thecamera. The dynamic range of light external to a vehicle exceeds 120decibels. When a car is driving at night, for example, very little lightmay be available whereas when driving in a bright sunlight, the lightintensity can overwhelm most cameras. Additionally, the camera must beable to adjust rapidly to changes and light caused by, for example, theemergence of the vehicle from a tunnel, or passing by other obstructionssuch as trees, buildings, other vehicles, etc. which temporarily blockthe sun and cause a strobing effect at frequencies approaching 1 kHz.

Improvements have been made to CMOS cameras that have significantlyincreased their dynamic range. New logarithmic high dynamic rangetechnology such as developed by IMS Chips of Stuttgart, Germany, is nowavailable in HDRC (High Dynamic Range CMOS) cameras. This technologyprovides a 120 dB dynamic intensity response at each pixel in amonochromatic mode. The technology thus has a 1 million to one dynamicrange at each pixel. This prevents blooming, saturation and flaringnormally associated with CMOS and CCD camera technology. This solves aproblem that will be encountered in an automobile when going from a darktunnel into bright sunlight.

There is also significant infrared radiation from bright sunlight andfrom incandescent lights. Such situations may even exceed the dynamicrange of the HDRC camera and additional filtering including polarizingfilters may be required. Changing the bias on the receiver array, theuse of a mechanical iris, light valve, or electrochromic glass or liquidcrystal or a similar filter can provide this filtering on a global basisbut not at a pixel level. Filtering can also be used with CCD arrays,but the amount of filtering required is substantially greater than thatrequired for the HDRC camera.

Liquid crystals operate rapidly and give as much as a dynamic range of10,000 to 1 but may create a pixel interference effect. Electrochromicglass operates more slowly but more uniformly thereby eliminating thepixel effect. This pixel effect arises whenever there is one pixeldevice in front of another. This results in various aliasing, Moirépatterns and other ambiguities. One way of avoiding this is to blur theimage. Another solution is to use a large number of pixels and combinegroups of pixels to form one pixel of information so that the edges andblurred and eliminate some of the problems with aliasing and Moirépatterns. Finally, range gates can be achieved as high speed shutters bya number of devices such as liquid crystals, garnet films, Kerr andPockel cells or as preferred herein as described in patents and patentapplications of 3DV Systems Ltd., Yokneam, Israel including U.S. Pat.No. 6,327,073, U.S. Pat. No. 6,483,094, US2002/0185590, WO98/39790,WO97/01111, WO97/01112 and WO97/01113.

One straightforward approach is the use a mechanical iris. Standardcameras already have response times of several tens of millisecondsrange. They will switch, for example, at the frame rate of a typicalvideo camera (1 frame=0.033 seconds). This is sufficiently fast forcategorization but probably too slow for dynamic object positiontracking when the object in the blind spot is traveling at a high speedrelative to the host vehicle.

An important feature of the IMS Chips HDRC camera is that the fulldynamic range is available at each pixel. Thus, if there are significantvariations in the intensity of light within the vehicle blind spot, andthereby from pixel to pixel, such as would happen when sunlight streamsand through a row of trees, for example, the camera can automaticallyadjust and provide the optimum exposure on a pixel by pixel basis. Theuse of the camera having this characteristic is beneficial to theinvention described herein and contributes significantly to systemaccuracy. CCDs generally have a rather limited dynamic range due totheir inherent linear response and consequently cannot come close tomatching the performance of human eyes.

A key advantage of the IMS Chips HDRC camera is its logarithmic responsethat comes closest to matching that of the human eye. One problem with alogarithmic response is that the variation in intensity from pixel topixel at an edge may be reduced to the point that the edge is difficultto recognize. A camera with less dynamic range can solve this problem atthe expense of saturation of part of the image. One solution is to takeseveral images at a different exposures and combine them in such amanner as to remove the saturation and highlight the edges. This isdescribed in the article “High Dynamic Range Imaging: Spatially VaryingPixel Exposures” referenced above.

Other imaging systems such as CCD arrays can also of course be used withthis invention. However, the techniques will be different since thecamera is very likely to saturate when bright light is present and torequire the full resolution capability when the light is dim. Generally,when practicing this invention, the blind spots will be illuminated withspots or a line of infrared radiation in a scanning mode. If a non-highdynamic range imager is used, the full illumination of the blind spotarea may be required. If eye-safe IR is used, then a broaderillumination beam is possible that in some cases can reduce or eliminatethe need for scanning.

In a preferred embodiment, infrared illumination is used although thisinvention is not limited to the use of infrared illumination. However,there are other bright sources of infrared that must be accounted for.These include the sun and any light bulbs that may be present outsidethe vehicle including headlights from other vehicles. This lack of ahigh dynamic range inherent with the CCD technology essentially requiresthe use of an iris, liquid crystal, light valve and/or electrochromicglass or similar filter to be placed between the camera and the scene.

Even with these filters however, some saturation can take place with CCDcameras under bright sun or incandescent lamp exposure. This saturationreduces the accuracy of the image and therefore the accuracy of thesystem. In particular, the training regimen that must be practiced withCCD cameras is more severe since all of the saturation cases must beconsidered because the camera is unable to appropriately adjust. Thus,although CCD cameras can be used, HDRC logarithmic cameras such asmanufactured by IMS Chips are preferred in many implementations. HDRClogarithmic cameras not only provide a significantly more accurate imagebut also significantly reduce the amount of training effort andassociated data collection that must be undertaken during thedevelopment of the neural network algorithm or other computationalintelligence system. Note that in some applications, it is possible touse other more deterministic image processing or pattern recognitionsystems than neural networks such as optical correlation techniques.

Another important feature of the HDRC camera from IMS Chips is that theshutter time for at least one model is constant at less than about 100ns irrespective of brightness of the scene. The pixel data arrives atconstant rate synchronous with an internal imager clock. Random accessto each pixel facilitates high-speed intelligent access to any sub-frame(block) size or sub-sampling ratio and a trade-off of frame speed andframe size therefore results. For example, a scene with 128 K pixels perframe can be taken at 120 frames per second, or about 8 milliseconds perframe, whereas a sub-frame can be taken at as high as 4000 frames persecond with 4 K pixels per frame. This combination allows the maximumresolution for the identification and classification part of the objectsensing problem while permitting a concentration on those particularpixels which track the leading edge of the object for dynamic positiontracking. In fact, the random access features of these cameras can beused to track multiple parts of the image and thus, in some cases,multiple objects simultaneously while ignoring the majority of theimage, and do so at very high speed.

For example, several motorcycles or pedestrians in the blind spot can betracked simultaneously by defining separate sub-frames for eachindividual object that is not connected to other objects. This randomaccess pixel capability, therefore, is optimally suited for recognizingand tracking multiple objects in a blind spot. It is also suited formonitoring the environment outside of the vehicle other than for thepurpose of blind spot detection such as collision avoidance andanticipatory sensing. Photobit Corporation of 135 North Los Robles Ave.,Suite 700, Pasadena, Calif. 91101 manufactures another camera with somecharacteristics similar to the IMS Chips camera. Other competitivecameras can be expected to appear on the market.

Photobit refers to their Active Pixel Technology as APS. According toPhotobit, in the APS, both the photo detector and readout amplifier arepart of each pixel. This allows the integrated charge to be convertedinto a voltage in the pixel that can then be read out over X-Y wiresinstead of using a charge domain shift register as in CCDs. This columnand row addressability (similar to common DRAM) allows for window ofinterest readout (windowing) which can be utilized for on chipelectronic pan/tilt and zoom. Windowing provides added flexibility inapplications, such as disclosed herein, needing image compression,motion detection or target tracking.

At least one model of the APS utilizes intra-pixel amplification inconjunction with both temporal and fixed pattern noise suppressioncircuitry (i.e., correlated double sampling), which produces exceptionalimagery in terms of wide dynamic range (˜75 dB) and low noise (˜15 e-rmsnoise floor) with low fixed pattern noise (<0.15% sat). Unlike CCDs, theAPS is not prone to column streaking due to blooming pixels. This isbecause CCDs rely on charge domain shift registers that can leak chargeto adjacent pixels when the CCD register overflows. Thus, bright lights“bloom” and cause unwanted streaks in the image. The active pixel candrive column buses at much greater rates than passive pixel sensors andCCDs.

On-chip analog-to-digital conversion (ADC) facilitates driving highspeed signals off chip. In addition, digital output is less sensitive topickup and crosstalk, facilitating computer and digital controllerinterfacing while increasing system robustness. A high speed APSdeveloped for a custom binary output application produced over 8,000frames per second, at a resolution of 128×128 pixels. It is possible toextend this design to a 1024×1024 array size and achieve greater than1000 frames per second for machine vision. All of these features areimportant to many applications of this invention.

U.S. Pat. No. 5,471,515 provides additional information on the APScamera from Photobit. To put this into perspective, a vehicle passinganother vehicle at a relative velocity of 60 mph moves approximately 1inch per millisecond relative to the slower vehicle. This renders theframe rate and computational times critically important and within thecapabilities of the HDRC and APS technologies.

These advanced cameras, as represented by the HDRC and the APS cameras,now make it possible to more accurately monitor the environment in thevicinity of the vehicle. Previously, the large dynamic range ofenvironmental light has either blinded the cameras when exposed tobright light or else made them unable to record images when the lightlevel was low. Even the HDRC camera with its 120 dB dynamic range may bemarginally sufficient to handle the fluctuations in environmental lightthat occur. Thus, the addition of an electrochromic, liquid crystal,light valve or other similar filter may be necessary. This isparticularly true for cameras such as the Photobit APS camera with its75 dB dynamic range.

At about 120 frames per second, these cameras are adequate for caseswhere the relative velocity between vehicles is low. There are manycases, however, where this is not the case and a higher monitoring ratemay be required. This occurs for example, in collision avoidance andanticipatory sensor applications as well as in blind spot applicationswhere one vehicle is overtaking another at high speed. The HDRC camerais optimally suited for handling these cases since the number of pixelsthat are being monitored can be controlled resulting in a frame rate ashigh as about 4000 frames per second with a smaller number of pixels.

Another key advantage of the HDRC camera is that it is quite sensitiveto infrared radiation in the 0.8 to 1 micrometer wavelength range. Thisrange is generally beyond visual range for humans thereby permittingthis camera to be used with illumination sources that are not visible tothe human eye. This IR sensitivity can be increased through special chipdoping procedures during manufacture. A notch frequency filter isfrequently used with the camera to eliminate unwanted wavelengths. Thesecameras are available from the Institute for Microelectronics (IMSChips), Allamndring 30a, D-70569 Stuttgart, Germany with a variety ofresolutions ranging from 512 by 256 to 720 by 576 pixels and can becustom fabricated for the resolution and response time required.

FIG. 2 illustrates the arrangement of FIG. 1 in a traffic situation.Optical assembly 6 on the subject or “host” vehicle contains anilluminating light source and a CMOS array. The illuminating lightsource of the optical assembly 6, either an array of scanning LEDs, ascanning laser radar device, laser spotlight or other source disclosedherein, distributes infrared radiation or energy in any appropriate formsuch as in the form of distinct narrow angle beams or a line that coversor fills in the blind spot between bounding lines 10 and 11. Any objectsuch as vehicle 23 that is within this blind spot will be illuminated byinfrared and the image of object will be captured by the CMOS array ofthe optical assembly 6.

An optical infrared transmitter and receiver assembly is shown generallyat 7 in FIG. 3A and is mounted onto the side rear view mirror 16.Assembly 7, shown enlarged, comprises a source of infrared radiationincluding an array of 20 infrared LEDs, for example, or one or morelaser diodes, shown generally at 13, and a CCD or CMOS array 14 oftypically 160 pixels by 160 pixels or more. The CCD or CMOS array 14 ishorizontally spaced apart from the LED array 13. In this embodiment, a“heads-up” display can be used to show the driver an artificial imageincluding the host vehicle and objects in the blind spot as describedbelow.

If two spaced-apart CCD arrays are used (e.g., array 14 and array 15shown in FIG. 3A), then the distance to the various objects within theblind spot can be found by using a triangulation algorithm that locatessimilar features on both images and determines their relative locationon the images. This is frequently referred to as a stereoscopic systemsuch as described in European Patent Application No. EP0885782 A1. Analternate method is to use a lens with a short focal length. In thiscase, the lens is mechanically focused to determine the clearest imageand thereby obtain the distance to the object. This is similar tocertain camera auto-focusing systems such as one manufactured by Fuji ofJapan. Other methods can be used as described in the patents and patentapplications referenced above.

FIG. 3B shows a similar arrangement for mounting on a truck mirror. Inthis case, since the geometry of the mirror provides greater separationvertically than horizontally, the illumination source 13 is placed onthe top of the mirror housing 27 and the imager 14 is placed at thebottom of the mirror housing 27. The “imager” 14 may comprise a CCDarray or CMOS array. Two or more spaced-apart imagers 14 can be used inthis embodiment as well and the techniques described above applied todetermine the relative location of features in images obtained by theimagers 14.

Once a vehicle exterior monitoring system employing a sophisticatedpattern recognition system, such as a neural network or opticalcorrelation system, is in place, it is possible to monitor the motionsof the object over time, and thereby determine if the object is actingin a predictable manner. If not, the driver of the host vehicle can bewarned so that he or she can take evasive action. For example, a vehiclemay be in the blind spot and the driver may be losing control of thevehicle as may happen in a passing situation when the passing vehiclehas hit a patch of ice. This warning may be sufficient to allow thedriver of the host vehicle to slow down and thereby avoid an accidentwith the out-of-control vehicle.

The system can also be used to turn on the vehicle hazard lights, soundthe horn or take other appropriate action in case the driver of thethreatening vehicle has fallen asleep and to warn other adjacentvehicles of a potentially dangerous situation. Thus, in general, anothervehicular system can be controlled based on the determination of thepresence and/or motion of the object detected in the blind spot. The useof a heads-up display is particularly useful for such a warning systemsince the driver is presumably looking through the windshield.Out-of-control monitoring can also apply to the host vehicle if itstrajectory is unexpected relative to objects along the roadside or otherproximate vehicles.

Infrared waves are shown coming from side transducer assemblies 17 and18 in FIG. 4. As such, assemblies 17, 18 constitute infraredtransmitters. In this case, CMOS imagers 19 and 20 are mounted on theside rear view mirrors providing ample displacement for triangulationcalculations. Note that in some embodiments, a wide angle view can beimplemented covering up to and exceeding 180 degrees for any of theimagers 17, 18, 19, 20 as well as others disclosed herein whereapplicable. Thus, FIG. 4 shows one arrangement of non-collocatedtransmitters and receivers, it being understood that other arrangementsin which the transmitters are not collocated with the receivers are alsowithin the scope and spirit of the invention. Additionally, FIG. 4illustrates alternate mounting locations for blind spot surround vehiclemonitoring in the corner light fixtures on the vehicle, the locationsbeing designated 61, 62, 63, 64. Each such location 61, 62, 63, 64, canprovide up to 270 degrees of view of the area surrounding the vehicle.Generally, the transmitter/illuminator and receiver will be locatedadjacent each other at these mounting locations but this need not be thecase and the transmitter/illuminator can be located above or below or atsome other convenient location relative to the receiver.

FIG. 5 illustrates two optical systems each having a source of infraredradiation and a CCD or CMOS array receiver. In this embodiment,transducer assemblies 5 and 6 are CMOS arrays having 160 by 160 or morepixels covered by a lens. The lens is carefully designed so that itcompletely covers the blind spot area under surveillance. One suchsensor placed by the left outside mirror where it can monitor the entirevehicle left exterior blind spot with sufficient resolution to determinethe occupancy of the blind spot. CCD's such as those used herein areavailable from Marshall Electronics Inc. of Culver City, Calif. andothers.

The lens need not be non-distorting. The distortion of a lens can bedesigned by modifying the shape of the lens to permit particularportions of the exterior of the passenger compartment to be observed.The particular lens design will depend on the location on the vehicleand the purpose of the particular receiver. In this example, the lightsource, which can be an array of modulated LEDS is collocated with theCMOS imager. Note that although only four beams are illustrated on eachside of the vehicle, typically twenty such beams are used. A modulatedscanning laser or laser spotlight can alternately be used.

CCD arrays are in common use in television cameras, for example, toconvert an image into an electrical signal. For the purposes herein, aCCD will be used interchangeably with CMOS and will be defined toinclude all devices, including CMOS arrays, TFA arrays, focal planearrays, artificial retinas and particularly HDRC and APS arrays, whichare capable of converting light frequencies, including infrared, visibleand ultraviolet, into electrical signals. The particular CCD array usedfor many of the applications disclosed herein is implemented on a singlechip that is less than two centimeters on a side. Data from the CCDarray is digitized and sent serially to an electronic circuit (at timesdesignated 12 herein) containing a microprocessor for analysis of thedigitized data. In order to minimize the amount of data that needs to bestored, initial processing of the image data can take place as it isbeing received from the CCD array. In some cases, some image processingcan take place on the chip such as described in the Kage et al.artificial retina article referenced above.

One method of determining distance to an object directly withoutresorting to range finders, requiring multiple arrays, is to use amechanical focusing system. However, the use of such an apparatus iscumbersome, expensive, and slow and has questionable reliability. Analternative is to use the focusing systems described in U.S. Pat. No.5,193,124 and U.S. Pat. No. 5,003,166. However, such systems can requireexpensive hardware and/or elaborate algorithms and again are slow.

Another alternative is where an infrared source having a widetransmission angle such that the entire contents of the blind spotilluminated, a sort of infrared floodlight or spotlight which preferablyis in the eye-safe IR frequency range. The receiving CCD transducers canbe spaced apart so that a stereographic analysis can be made by thecontrol circuitry 12. This circuitry 12 contains a microprocessor withappropriate pattern recognition algorithms along with other circuitry asdescribed above. In this case, the desired feature to be located isfirst selected from one of the two returned images from either of theCCD transducers. The software then determines the location of the samefeature, through correlation analysis or other methods, on the otherimage and thereby, through analysis familiar to those skilled in theart, determines the distance of the feature from the transducers.

Transducer assemblies 5 and 6 are illustrated mounted onto the sidemirrors of the vehicle, however, since these transducers are quitesmall, typically approximately 2 cm on a side, they could alternately bemounted onto the side of the vehicle or many other locations whichprovide a clear view of the blind spot.

A new class of laser range finders has particular application here. Thisproduct, as manufactured by Power Spectra, Inc. of Sunnyvale, Calif., isa GaAs pulsed laser device which can measure up to 30 meters with anaccuracy of <2 cm and a resolution of <1 cm. This system can beimplemented in combination with transducer assemblies 5 or 6. Once aparticular feature of an object in the blind spot has been located, thisdevice can be used in conjunction with an appropriate aiming mechanismto direct the laser beam to that particular feature. The distance tothat feature is then known to within about 2 cm and with calibrationeven more accurately.

In addition to measurements within the blind spot, this device hasparticular applicability in anticipatory sensing applications exteriorto the vehicle. An alternate technology using range gating or phasemeasurements to measure the time-of-flight of electromagnetic pulseswith even better resolution can be implemented based on the teaching ofthe McEwan patents or the Intelligent Technologies Int'l patentapplication listed above or by modulation of the laser beam and usingphase measurements such as disclosed in U.S. Pat. No. 5,653,462.

FIG. 6A is an overhead view and FIG. 6B a side view of a truck showingsome preferred mounting locations of optical exterior vehicle monitoringsensors (transmitter/receiver assemblies or transducers) 160, 161. In atypical device, the diameter of the lens is approximately 2 cm and itprotrudes from the mounting surface by approximately 1 cm. This smallsize renders these devices almost unnoticeable by observers exterior tothe vehicle.

Since the sensors and transducer assemblies used in some embodiments ofthe invention are optical, it is important that the lens surface remainsrelatively clean. Control circuitry 120 contains a self-diagnosticfeature where the image returned by a transducer assembly or sensor iscompared with a stored image and the existence of certain key featuresis verified. If a receiver part of an assembly or sensor fails thistest, a warning is displayed to the driver that indicates that cleaningof the lens surface is required.

The truck system shown in FIGS. 6A and 6B illustrates the use of asingle blind spot detection system for the entire length of truck. Thefundamental issue that determines the size of the blind spot that can bemonitored with a single system relates to the ability to measure thelocation of the object. When a HDRC camera is used, if an object canseen in the blind spot by the human eye, then the camera should also beable to obtain a reasonable image. At night, this would require that theobject in blind spot have some form of attached illumination. On a darkcloudy night, the human eye has trouble seeing a car parked along theroadway with its lights extinguished. The more distant the object, themore difficult it is to obtain a recognizable image if illumination isnot present.

A significant improvement to the situation occurs if the blind spot isflooded even with low-level infrared radiation. This argues for aninfrared floodlight in addition to the distance measuring infraredsystem. If an infrared floodlight is used along with multiple camerasdisplaced from one another, then the location of object in the blindspot can be determined by optical correlation between the two images andby triangulation calculations. This may be a practical solution fortrucks especially those containing multiple trailers. A truck withmultiple cameras placed along the left side of the vehicle isillustrated in FIG. 6C. Of course, a bright IR floodlight, preferably inthe eye-safe part of the spectrum, based on a high powered diode laserand appropriate optics can be used.

The other limiting case is when bright sunlight is present and only asingle imager is used for a particular blind spot. For this case, ascanning laser infrared beam, or a high powered laser diode spotlight,can still be distinguished as a reflection off of an object in the blindspot providing a narrow notch filter is used to eliminate allfrequencies other than the particular infrared frequency used. Even inthis case, the distance where the reflected infrared beam can beascertained in bright sunlight can be limited to perhaps fifteen meters.Therefore, this system can be marginal for long trucks unless multiplesystems are used along the side of the truck as shown at 28 in FIG. 6C.Note that certain mid-infrared frequencies having wavelengths above 10microns are particularly good in that the radiation from the sun issignificantly attenuated. Low cost imagers are not currently availablebut are under development for these frequencies.

From the above discussion, it would appear that multiple cameras may bethe only viable solution for long trucks. A further problem arises inthis system design in that if the cameras are located on differenttrailers, or for some other reason can move relative to each other, thenthe analysis computer must know the location and orientation of eachcamera. There are a variety of ways of accomplishing this orientationsuch as through locating laser beams or monitoring the relativepositions of the various components of the truck. In one example, alaser beam is used to illuminate a spot on the road that can be observedfrom multiple camera locations. Using the position of this reflected dotin the images acquired by various cameras, the relative orientation isapproximately determined. Naturally, more complicated and sophisticatedsystems are possible. RFID tags offer another method of determining therelative location of a point on a trailer relative to the tractor ifmultiple antennas are used and if the relative time of arrival of thereceived RFID signals are measured.

The blind spot monitoring systems described above in FIGS. 6A, 6B and 6Care mainly applicable for blind spots occurring during highway travel.For urban travel of a truck where frequent turns are made, another blindspot occurs on right hand side of the vehicle (in countries wherevehicles drive on the right side of the road) and extends somewhatforward of the vehicle and back somewhat beyond vehicle cab. This area,which cannot be seen by the driver, can contain pedestrians, smallvehicles, bicycles, curbs, fire hydrants, motorcycles, as well as avariety of other objects. Another more local blind spot system thatcovers this area is therefore necessary, as illustrated in FIGS. 7A and7B and which is designated 24.

The applications described herein have been illustrated mainly using thedriver side of the vehicle. The same systems of determining the positionof an object in the blind spot are also applicable on the passengerside.

A significant number of children are killed every year by being run overby school buses. This tragic accident occurs when a child leaves theschool bus and walks in front of the bus in the driver's blind spot. Thedriver starts driving the bus and strikes the child. A blind spotmonitor of this invention, i.e., one or more transducer assemblies, isshown mounted on the front of school bus 25 near the top of the enginecompartment 180 in FIGS. 8A and 8B. This monitoring system alerts thedriver of the presence of an object obstructing the path of the schoolbus 25.

The system shown in FIG. 9 illustrates a blind spot monitoring system 26built according to the teachings of this invention. The system canutilize a high dynamic range camera, identification and rangingcapability with or without illumination or, alternately, a linearscanning laser range meter or laser spotlight. The view provided to thedriver shows the location, size and identity of all objects that arewithin the path of the backing vehicle. The display provides maximumcontrast by using icons to represent the host vehicle and the objects inthe blind spot. Although this is shown for a truck, it is equallyapplicable for other vehicles including buses and automobiles. It canalso be used in a rear impact anticipatory sensor where both thedisplacement and velocity, by either Doppler or differencing distancemeasurements, of the approaching object can be determined.

The monitoring system 26 could be activated whenever the vehicle is inreverse, unless it is also used for rear impact anticipatory sensing.Thus, the system 26 would not be needed when the vehicle is travelingforward. When the gear is shifted into reverse, a sensor could beprovided to detect the change in gear and then activate the monitoringsystem 26. Similarly, a monitoring system which is for forward blindspot such as in front of the bus 25 shown in FIGS. 8A and 8B could bedesigned to activated only when the vehicle is in forward gear and notstopped or in reverse. As such, when the gear is shifted into forward,the system 25 would be activated.

In FIGS. 5-9, the illumination is shown as discrete bands. This need notbe the case and, as discussed above, sources of general illumination orscanning systems can be used.

If both a forward and rear monitoring system are provided, theactivation of both of these monitoring systems would not need to besimultaneous but could depend, e.g., on the direction of travel of thevehicle. In this case, a single display could be provided to the driverand alternatively display the contents of the forward blind spot or rearblind spot depending on the direction of the travel of the vehicle,i.e., in which gear the vehicle is in.

FIG. 10 illustrates a block diagram showing interface between five blindspot monitoring systems and control circuitry 12. The control circuitry12 monitors the output from the five blind spot monitoring systems andcreates icons and places the icons on a display 30,31 that shows thehost vehicle and all objects in the immediate vicinity of the hostvehicle. Software is provided in the microprocessor to sound a warningsignal or activate haptic actuators 33 under a predetermined set ofcircumstances such as an attempt by the driver to change lanes into alane occupied by an object in the blind spot. This warning signal mayalso be activated if the driver activates the turn signal. In additionto the audio warning signal, a visual flashing signal provided on thedisplay and a vibration or pressure or torque or other haptic signalapplied to the steering wheel to prevent or make it more difficult fordriver execute the maneuver.

The display 30, 31 would selectively or alternatively display thecontents of each blind spot. A screen-within-a-screen type display couldalso be used to display one blind spot in a majority of the screen andanother blind spot in a small portion of the screen. As noted above, theblind spot displayed could depend on the status of the gear of thevehicle. The blind spot displayed could also depend on the direction ofturning of the front wheels, the direction of turning of the rear wheelsand/or the activation of the right or left turn signals.

External monitoring so far has been concerned with a host or residentvehicle monitoring the space in its environment. Naturally, there arevehicles that precede the host vehicle and experience the sameenvironment prior to the host vehicle. Information from such vehicles,which can be called “probe” vehicles, can be communicated to the hostvehicle to aid that vehicle in its safe travel. This brings up thesubject of communication between vehicles which is covered in otherpatents and patent applications assigned to ATI and ITI and incorporatedby reference herein so the subject will only be briefly discussed here.Generally, communication between vehicles is composed of that whichshould be transmitted in the most expedient fashion to aid in collisionavoidance and that where some delay can be tolerated. For the firsttype, a broadcast protocol is preferred where each vehicle transmits amessage to surrounding vehicles directly and without employingnetworking protocols, error correction, handshaking etc. When manyvehicles are so broadcasting, the host vehicle needs to have a method ofdetermining which vehicle to listen to which can be done, for example,by a CDMA system where the code is a function of the transmittingvehicle's location such as its GPS coordinates. The receiving vehiclewith a resident map can determine the codes where potentiallythreatening vehicles are resident and listen only to those codes. Forthe second type of communication, the Internet or similar ubiquitoussystem is suggested. Each probe vehicle would communicate informationsuch as the existence of a new construction zone, a patch of ice, fog orother visibility conditions, an accident or any other relevantinformation to a central source which would monitor all suchtransmissions and issue a temporary map update to all vehicles in thevicinity over the internet, or equivalent. If the probe vehicle came onan accident, then such a vehicle may also transmit a picture(s) of theaccident to the central control station. This picture(s) could betransmitted automatically without any action of the driver who may noteven be aware that it is occurring. The central control station couldthen determine the nature, seriousness, extent etc. of the accident andissue a meaningful update to the map of the area and later remove theupdate when the accident is cleared. This will permit the display of theaccident on a map display of equipped vehicles.

This idea can be extended to cover other hazards. If some probe vehiclesare equipped with appropriate sensors such as radiation, chemical and/orbiological sensors, an early warning of a terrorist attack can betransmitted to the central control station all without any action on thepart of the vehicle operator. In general, any information that can besensed by a vehicle traveling on a roadway, including the maintenancestate of the roadway itself, can be automatically monitored and relevantinformation can be transmitted automatically over the Internet, orequivalent, to a control station along with appropriate pictures ifavailable. Any equipped vehicle can be a probe vehicle.

This assumes the existence of ubiquitous Internet, or equivalent. Thisis by far the least expensive way of providing such a capability to the4 million miles of roads in the continental United States. Proposals arenow being considered to put transceivers every 100 meters along themajor highways in the US at a cost for installation of billions ofdollars. Such transceivers would only cover the major highways eventhough the majority of fatal accidents occur on other roadways. Themaintenance cost of such a system would also be prohibitive and itsreliability questionable. For far less money, the continental US can becovered with IEEE 802.11 based systems or equivalent. Such a transceivercan cover up to a radius of 30-50 miles thus requiring onlyapproximately 500 to 1000 such stations to cover the entire continentalUS. Naturally, more units would be required in densely populated areas.The cost of such units can be as low as a few thousand dollars each buteven if they cost a million dollars each, it would be a small costcompared with the alternative roadside transceivers.

Initially some areas of the country will not have such 802.11 orequivalent stations. For those areas, map updates can be transmitted bya variety of methods including a station on satellite radio or someother satellite transmitting system, through the cell phone network orany other existing or special communication system. If the selectedsystem does not support two way communication, then the messages createdby the probe vehicle can be stored and transmitted when access to theInternet is available. A probe vehicle can be a specially equippedvehicle or all vehicles with the appropriate equipment.

Eventually, all cars will be connected with a combination of broadcastsystem for collision avoidance and ubiquitous Internet connections formap-based road hazards that are discovered by the vehicle. As a vehicletravels down a road and discovers an accident for example, a photographof that accident will be stored and uploaded to the Internet forinterpretation by a human operator who will then download a messagebased on the map location of the accident to warn other vehicles thatare in the vicinity until the accident is cleared up which can bedetermined by another probe vehicle.

When all cars have the system, there will be much less need for surroundvehicle monitoring except for to search for bicycles, perhapsmotorcycles, pedestrians, animals, land slides, rocks, fallen trees,debris etc. All other vehicles will be properly equipped and the RtZF®can be on special lanes that permit autonomous vehicles.

There should not be any obstacles on the highway and when one isdiscovered, it should be photographed and uploaded to the internet forproper handling in terms of warnings and removal of the hazard. Untilthe time comes when the Internet is everywhere, alternate systems canpartially fill in the gaps such as XM radio and other satellite basedsystems. This would be used only for downloading map changes. Foruploading information, the vehicles would wait, maintaining data to besent to a database until they have a direct internet connection.

To achieve ubiquitous internet coverage, IEEE 802.11 or Wi-Fi stations(or WiMAX or WiMobile or equivalent) would be placed around the nation.If, for example, each station had a radial range of 30-50 miles or morethen approximately 500 to 1000 such stations could be strategicallyplaced to provide nationwide coverage. It is anticipated that the rangeof such stations will be substantially increased but that the number ofrequired stations will also increase as usage of the ubiquitousinternet, or equivalent, network also increases. In that case, privateindustry can be earning revenues through non-safety use access charges.An estimate of the cost of a typical station is between $10,000 and$100,000 most of which is for the land and installation. The total costthus would be around a maximum of $100 million which is a small fractionof the multi-billion dollar estimate by the Federal Highway Departmentto implement their proposed DSCR system with transceivers every 100meters along the Federal Highway System, a system that would leave mostof the nation unprotected and in general be of marginal value. There aremany towers in place now for use by radio and TV stations and cellulartelephones. It is expected that such towers can also be used for thisubiquitous network thus reducing the installation costs.

Such a proposed system could also broadcast a timing signal as well asthe differential corrections to support Differential GPS (DGPS). Itcould even broadcast a GPS-type signal and thus eliminate the dependenceof the RtZF® system on GPS. In other words, anyone could obtaincentimeter level position accuracy without GPS. This concept may requirea mapping of multipath delays in some urban areas.

Such a ubiquitous internet system could also provide continuous trafficupdates, route guidance information as well as weather information,automatic collision notification, diagnostic and prognostic telematicscommunications to the manufacturer and dealer etc., and in fact alltelematics transmissions would be easily achieved with such an internetsystem.

Looking further, ubiquitous internet could eliminate all communicationsystems that are currently used in the US including radio, TV, Cellularphones, XM radio and all satellite communications, telephone, OnStar®and all telematics, DSRC. Everyone could have one phone number and onephone that would work everywhere.

Other applications include remote sensing applications for home and boatsecurity and homeland security applications, for example. Any point onthe continental US would be able to communicate with the internet. Ifthis communication happens only occasionally, then the power can beminimal and can be boosted by some form of energy harvesting and thussuch a sensor could operate from years to infinitely without a powerconnection from batteries. For example, all monitoring and trackingoperations that require satellite communication such as disclosed inU.S. patent application Ser. No. 10/940,881 entitled Asset SystemControl Arrangement and Method, could be handled without satellitecommunication for the continental United States.

1.6 Lane Departure Warning System

Systems are now appearing on the market that warn of an unintended lanedeparture. Such systems are not based on the flash lidar systemsdisclose herein and then are seriously degraded by weather situationsand particularly degraded when the roadway is wet or covered with snow.Such systems can be improved using the teachings herein through the useof pattern recognition and particularly neural networks and Kalmanfilters as well as other image analysis technologies disclosed hereinincluding the lidar systems discussed above. Generally, the inventor ofthe inventions herein believes that the preferred approach at road andlane departure warning systems are those that are based on accurate mapsand vehicle accurate location systems such as DGPS. In the meantime,before such systems are in place, use of the imaging and illuminationsystems disclosed herein provides an improvement to the state of the artof vision-based lane and road departure systems.

1.7 Night Vision

Various vehicle manufacturers are offering a night vision system on somevehicle models. Some of these systems are based on passive infraredradiation that is naturally emitted from warm bodies and is in the longwave or thermal region of the IR spectrum. Other manufacturers areoffering active IR systems that operate in the near IR region of thespectrum that is just below the visual band in frequency. Despite claimsto the contrary, it is the view of the inventor herein that such systemsare of marginal value and may even contribute to degrading the safety ofthe vehicle since they can act as a distraction. A preferred approach,in the opinion of the inventor herein, is to use active IR systems asdisclosed above and then to analyze the received images to identifyobjects that may be of interest to the vehicle operator. Once such anobject is identified, a deer or pedestrian for example, an icon isplaced on a heads-up display on the windshield where the operator wouldsee the object if he or she could see it. This requires knowledge of thelocation of the eyes of the operator which can be obtained from suitableoccupant sensors disclosed in other of the current assignee's patentsand patent applications.

1.8 Headlight Control

A rather complicated approach to automatically dimming headlights isdisclosed in U.S. Pat. No. 6,587,573. The system includes a number ofapproximations and arbitrary decisions as to when to turn on the brightlights and when to switch to dim some based on a guess as to where avehicle that the host vehicle is passing is located. Many of theproblems which this patent attempts to solve disappear when all vehiclescan communicate their location to the host vehicle and thus the hostvehicle knows exactly when to dim its lights without any arbitrarydecisions. Similarly, when accurate maps are in every vehicle, theproblems with reflection from signs, confusion with street lights orhouse lights all also disappear. Thus the problem is reduced torecognizing the headlights and tail lights of vehicles that do not havethe complete collision avoidance system as described herein and inpatents assigned to ATI and ITI. In this interim situation, the hostvehicle needs to recognize a large number of image situations where suchvehicle lights are present within the images obtained by the vehicleexternal imaging system. This is a natural problem for neuralnetwork-based pattern recognition systems that can be trained onliterally millions of different images to permit the accuratedifferentiation of authentic vehicle lights from reflections and othertypes of lighting without going to the effort suggested by the '573patent. The headlight dimming system need not be a separate system butcan be incorporated in the general surround vehicle monitoring systemdisclosed herein. The '573 patent goes through enormous efforts to solveproblems that are very simple for the human vision system which showsthat the ultimate solution to this and many other problems lie in moresophisticated neo-cortex simulation programs. In the meantime, neuralnetwork and other pattern recognition techniques are a preferred methodof solving the headlight dimming problem.

Through the imaging system disclosed herein, an entire image of anapproaching vehicle can be obtained and identified which will also clearup confusion resulting from reflections from road surfaces or signs aswell as house and street lights. This will also work if the vehicleslights are not on or if they are defective. A common problem not solvedby the '573 patent which is solved herein, results from the case where avehicles lights blind the eyes of a truck driver in cases where ahighway separation blocks vehicle lights from automobile driver's eyesbut not that of a truck driver who sits much higher.

In one sense, location is the key to headlight control. If the vehicleis getting a reflection from someplace that is not on the road or wherea vehicle could not be, then the system can know that it is not from avehicle. Therefore, a key aspect of headlight control is having everyvehicle have an accurate map.

Even before maps are universally available, pattern recognition candistinguish between reflections off of a road surface or a sign fromanother vehicle by the shape of the reflection. If the system is trainedon what a reflection from a road or a hill is or the reflection from asign is, there should be no confusion between that and the headlights orthe taillights of a vehicle or with house or street lights or anythingelse, the only issue being accurate pattern recognition. Similarly, therain on the windshield or a dirty lens problems discussed in the '573patent should be easily detectable using pattern recognition. Thegeneral theory is that in contrast to the '573 patent, the properapproach is to put the intelligence into the pattern recognition systemand not in the deterministic programming which will always have holes.

2. Displays

An advanced display system can provide a simple icon image of the hostvehicle and all surrounding vehicles as viewed from above. In thismanner, with a simple glance, the driver can determine the location andidentity of all objects that are in his blind spot or in the vicinity ofthe vehicle in any direction. If this display is kept simple, then theproblems of visual dynamic range become much less severe. That is, ifthe driver need only see dark objects on a white background and if thesize of these objects is sufficient, then the display could be viewedboth at night and under daylight conditions.

In order to display objects on all sides of the vehicle, there shouldeither be a type of fisheye camera and lens set-up which can becentrally mounted on the vehicle roof or at least four wide anglecameras. The images from each of these cameras can be analyzed by apattern recognition system, such as one based of neural networks, toidentify objects in the field of view. Once these objects are identifiedand their position relative to the vehicle determined, then a displaycan be constructed from any viewpoint or camera focal length asdiscussed in EP1179958. Since that patent went into considerable detailillustrating how such images can be created from camera image data, asimilar discussion will not be presented here. All of the createdsynthetic views discussed in the '958 patent can be created using iconsto represent the objects that surround the vehicle. The views can bemade to zoom in or out to give different perspectives which may aid inparking or in heavy traffic or when the vehicle is traveling at highspeed. During such a zoom operation, only a portion of the host vehicleicon may be shown, for example when the vehicle is attempting to park.If color cameras are used, the icons can be colored to represent thecolor of the monitored vehicles or other objects. When an object isgetting dangerously close to the host vehicle or otherwise threatening,its icon can be intensified, made to flash or otherwise modified to callattention to the object.

If the vehicle knows its location and has accurate maps, the existenceof lane boundaries and all fixed objects can be accurately displayed asicons on the display. If a route guidance system is also present, allturns and other directions can be displayed also on a map display. Thepath that the vehicle should take can be displayed in yellow, forexample. Additionally, the path that the vehicle is projected to takewithin the next 30 seconds, for example, based on the steering wheelangle and vehicle velocity, can also be represented in a manner similarto that disclosed in U.S. Pat. No. 594,931.

In some parts of the U.S., satellite images are available in real timethat show traffic patterns including the subject vehicle. If the vehicleknows exactly its location and the location of the image, then a view ofthe area surrounding the subject vehicle can be superimposed on adisplay and the vehicle operator can see the traffic situationsurrounding his or her vehicle from above. Also, using patternrecognition, the salient features of the image can be extracted anddisplayed as icons on a display thereby simplifying the interpretationproblem for the driver as discussed above. However such a system wouldfail in tunnels and under heavy foliage.

A preferred embodiment of at least one of the inventions herein is touse an active or passive optical system for monitoring the presence ofobjects in the area of interest external to the vehicle. Patternrecognition technologies such as neural networks and optical correlationsystems will be used to positively identify the object that is in themonitored areas. This object may be a pedestrian, bicyclist,motorcyclist, guardrail, animal, automobile, truck, fire hydrant, tree,telephone pole, sign or whatever. The system will be trained orotherwise programmed to inform the operator either optically or orallythat such an object appears in the blind spot. It will also inform thedriver as to which blind spot contains the object. The system can alsoinform the driver as to whether this object is moving or stationary inan absolute sense and/or also in relation to the host vehicle. Thisinformation can be presented to the operator in a variety of ways.Initially, a light or simple icon can appear on the rear view mirror,for example, indicating either that some object exists or that aparticular object exists.

In more sophisticated systems, an icon representing the object can beplaced on a simple icon display which can show the vehicle from anoverhead view, or other convenient view, and an icon which shows theblind spot object and its location. Alternately, an oral annunciationcan be provided which tells the driver that, for example, there is aguardrail three feet to his left, or that there is an automobileapproaching from the rear in an adjacent lane at a relative speed of 100kph and is currently 50 feet behind the vehicle. All of these types ofwarnings can be provided if identification can be made of the object inthe blind spot and an accurate measurement made of the position andvelocity of that object relative to the host vehicle. This will bediscussed below.

It can be seen from this description that a system in accordance withthe invention will inform the driver of the type of object in the blindspot, where it is located specifically and/or what its velocity isrelative to the host vehicle, and in more sophisticated systems, showgraphically an icon showing the object relative to the vehicle from anoverhead view, for example, which is easily understandable by the driverwith a mere glance at the display. Therefore, the system overcomes allof the objections and problems described above with respect to the priorart systems.

Although simple icon displays are contemplated by this invention, thisis due to the lack of sophistication or capability of current displaytechnology. In other words, the dynamic range of light that can beemitted by conventional displays is insufficient to display other thanthe simplest messages. Technology advances, and it is expected thataccurate color displays with high dynamic range will be become availablebased, for example, on organic display technology (OLED). When suchdisplays are available, a more accurate representation of the object inthe blind spot even to the point of an actual image might becomefeasible.

The inventions herein do not generally contemplate the use of rear viewmirrors to permit the vehicle operator to actually see the contents ofthe blind spot. This is because to accurately accomplish this requiresknowledge of the position of the eyes of the driver. It is been observedthat drivers adjust side rear view mirrors over an extended range thatrenders the use of the mirror angle unsuitable for determining theposition of the driver's eyes. Furthermore, the driver may change his orher seating position without changing the position of the rear viewmirror. Occupant sensing systems are now being developed for vehiclesthat have the capability of determining the location of the eyes of avehicle operator. For those vehicles that contain such a system, thepossibility exists not only to automatically adjust the mirror tooptimally display contents of the blind spot, but also to change theorientation of the mirror when some object that the driver should beaware of is in the blind spot. This invention therefore contemplatessuch activities when occupant sensing systems are placed on vehicles.

FIG. 11 illustrates a control module 36 that contains a variety ofelectronic components 37-42. The control module is connected to theblind spot monitoring system including the transducer assemblies 5-9 bywires, not shown, or wirelessly and in turn it connects to a display onthe instrument panel 31 or a heads-up display 30. Based on thecalculations performed in a microprocessor 41, the control module 36creates the icons on displays 30 and 31 and additionally initiates audioand haptic warnings as described above. The connection between thecontrol module 36 and the audio and haptic actuators may be a wiredconnection or a wireless connection.

FIG. 12 is a further illustration of the heads-up display 30 shown inFIG. 11. The heads-up display 30 is constructed according to well-knownprinciples and the image is projected focused in front of vehicle 29such that the driver can observe the image without taking his or hereyes from the road.

Above, it has been assumed that data for determining the location ofvehicles that surround the host vehicles would come from cameras. Thisneed not be the case when vehicles are equipped with avehicle-to-vehicle communication system in which case, the host vehiclecan obtain the location and vehicle type information from transmissionsdirectly from another vehicle. One approach is to have each vehicleperiodically broadcast its ID, position and velocity. This could happenevery 10 milliseconds, for example. Such information could aid in theselection of the proper icon and the placement of that icon onto thedisplay. The channel selected for the broadcast communication could bebased on the location of the broadcasting vehicle. The host vehiclecould use a common algorithm plus a map to determine the channelselected by the broadcasting vehicle, and the area where the hostvehicle wishes to monitor, thereby greatly limiting the number ofvehicles that it would need to listen to. The broadcast approach coupledwith an icon display is applicable to vehicles of all types, andairplanes in particular.

Also, the displays discussed above have been assumed to be twodimensional. Three dimensional displays, especially for airplanes, arealso possible and the invention is not limited to two-dimensionaldisplays.

3. Identification

Use of trainable pattern recognition technologies such as neuralnetworks is an important part of some of the inventions disclosedherein, although other non-trained pattern recognition systems such asfuzzy logic, correlation, Kalman filters, template matching and sensorfusion can also be used. These technologies are implemented usingcomputer programs to analyze the patterns of examples to determine thedifferences between different categories of objects. These computerprograms are derived using a set of representative data collected duringthe training phase, called the training set. After training, thecomputer programs output computer algorithms containing the rulespermitting classification of the objects of interest based on the dataobtained after installation on the vehicle.

These rules, in the form of an algorithm, are implemented in the systemthat is mounted onto the vehicle. The determination of these rules isimportant to the pattern recognition techniques used in this invention.Neural networks either singularly or combination neural networks arecontemplated by some of the inventions disclosed herein. Combinationneural networks are groups of two or more neural networks and includemodular neural networks and ensemble neural networks among others. Alsoother forms of neural-based systems such as cellular neural networks andsupport vector machines are also contemplated by this invention.

Artificial neural networks using back propagation are thus far the mostsuccessful of the rule determination approaches. However, research isunderway to develop systems with many of the advantages of backpropagation neural networks, such as learning by training, without thedisadvantages, such as the inability to understand the network and thepossibility of not converging to the best solution. In particular, backpropagation neural networks will frequently give an unreasonableresponse when presented with data that is not within the training data.It is known that neural networks are good at interpolation but poor atextrapolation. A combined neural network fuzzy logic system, on theother hand, can substantially solve this problem. Additionally, thereare many other neural network systems in addition to back propagation.In fact, one type of neural network may be optimum for identifying thecontents of the blind spot and another for determining the location ofthe object dynamically.

The discussion thus far has identified pattern recognition systems andparticularly neural network pattern recognition systems to be used toidentify the contents of the blind spot. One particular neural networkarchitecture has been particularly successful in this field. This isknown as modular neural networks. The concept behind modular neuralnetworks is that when a complicated task is to be accomplished by aneural network, significant improvements in speed and accuracy cansometimes be obtained if the overall problem is divided into a number ofsmaller problems. A separate neural network is then assigned eachsub-task. Thus, a network of neural networks is created. An alternateand also successful approach is the use of Associative-Projective NeuralNetworks (APNN).

When a human observes a tree, the human mind concentrates oncharacteristics of that tree and not on characteristics of anautomobile. Thus, the human mind appears to operate also as a modularneural network. There are many ways of applying this concept to blindspot monitoring. Since both the identity and the location of object inthe blind spot are to be determined, it is logical to therefore separatethe problem into a first neural network that determines the identity ofthe object and then a variety of additional neural networks that, giventhe identity of the object, determine its location based of attributessuch as the size of the object in the image. In addition, a separateneural network may be trained to segregate any unknown objects from datathat are not understood by the neural networks because nothing similarwas a part of the training database.

Additional tasks that can be allocated to specific neural networks areto determine the environment that the vehicle is operating in.Obviously, an automobile in a blind spot looks considerably different atnight with its headlights on than in bright sunlight. The identificationand also the position determining tasks can be more accurate if they aresegregated by lighting conditions. Similarly, the presence of fog,smoke, rain, snow, soiled lenses, and other factors can have asignificant effect on the system accuracy and the determination of suchconditions can be allocated to separate groups of neural networks.

In some embodiments of this invention, the rules are sufficientlyobvious that a trained researcher can look at the returned opticalsignals and devise an algorithm to make the required determinations. Inothers, artificial neural networks are frequently used to determine therules. One such set of neural network software for determining thepattern recognition rules, is available from the NeuralWare Corporationof Pittsburgh, Pa. and another from International Scientific Research inPanama City, Panama. Numerous books and articles, including more than500 U.S. patents, describe neural networks in great detail and thus thetheory and application of this technology is well known and will not berepeated here. Neural networks are now beginning to gain more widespreaduse in the automotive industry including their use for engine control,occupant spatial sensing for the control of airbags, side and frontalcrash sensor algorithms and vehicle diagnostic systems.

The system generally used in this invention for the determination of thepresence of an object in the blind spot is the artificial neural networkor a neural-fuzzy system. In this case, the network operates on thereturned signals from the CCD or CMOS array as sensed by transducerassemblies such as 5, 6, 7, 8 and 9 in FIG. 1, for example. For the caseof the left blind spot, through a training session, the system is taughtto differentiate between many cases including automobiles, pedestrians,bicycles, trucks, animals, motorcycles, fences, guard rails, parkedvehicles etc. This is done by conducting a large number of experimentswhere data from each of these objects is captured in a variety ofpositions, velocities and vehicle operating conditions (rain, night,bright sunlight, rural roads, interstate highways, etc.). As many as1,000,000 such experiments can be run before the neural network issufficiently trained and validated so that it can differentiate amongthe various cases and output the correct decision with a very highaccuracy.

Once the network is determined, it is possible to examine the result todetermine, from the algorithm created by the neural network algorithmgenerating software, the rules that were finally arrived at by the trialand error training technique. In that case, the rules can then beprogrammed into a microprocessor. Alternately, a neural computer can beused to implement the network directly. In either case, theimplementation can be carried out by those skilled in the art of patternrecognition using neural networks.

Many systems are now on the market that monitor obstructions in the rearof a vehicle and warn the driver of the existence of such obstructionswhen the driver is backing a vehicle. The technologies currently usedinclude radar, ultrasound and TV cameras. Neither radar nor ultrasoundare generally capable of identifying the object and most such systemscannot locate the object which might allow the driver to slightly changehis or her direction and avoid a curb or pole, for example. Thetelevision camera systems typically do not have illumination sources andat best produce a poor television image to the driver that is difficultto see in sunlight.

FIGS. 13A, 13B and 13C illustrate one preferred method of separating anobject in the blind spot from other objects in preparation for inputinto a neural network for identification and/or position determination.FIG. 13A illustrates a view of the image as seen by a side rear viewtransducer assembly 7 of FIG. 1.

Various filters are employed to simplify and idealize the view theoutput of which is shown in FIG. 13B. A variety of technologies, such asdiscussed above, exist to eliminate remaining background objects andisolate the vehicle to arrive at the image as shown in FIG. 13C.

In one preferred method, the distance to the objects to the left andright of the vehicle can determined by the laser radar system describedabove. This permits the elimination of objects that are not in the sameplane as the blind spot vehicle. Any of the distance measuring schemesdescribed above along with pattern matching or pattern linkingtechniques can be used to extract the vehicle. Other techniques involvethe use of relative motion of the object in the blind spot that mayinvolve the use of optical flow calculations. A preferred method isthrough the use of range gating as discussed above. The goal is a fullthree-dimensional representation of entire scene or region on interest(ROI) has been achieved. Therefore, a variety of techniques can be useddepending on particular problem at hand.

FIG. 14 illustrates a lane-changing problem in congested traffic. Inthis illustration, the driver of vehicle 46 wants to change lanes topass vehicle 47. However, vehicle 45 is in the blind spot and if vehicle46 attempts this lane change, an accident may result. Using theteachings herein, the driver of vehicle 46 will be made aware eitherthrough a visual display or through warning signals, optical, audioand/or haptic, should the driver attempt to execute such a lane change.The driver may be made aware of the presence of the vehicle 45 in theblind spot upon activation of the turn signal, upon detection of thebeginning of the lane change as reflected in the turning of the steeringwheel or front wheels of the vehicle and/or by the presence of an iconshowing the vehicle 45 in the display 30, 31.

A detailed discussion of pattern recognition technology as applied tothe monitoring and identification of occupants and objects within avehicle is discussed in U.S. Pat. No. 5,829,782. Although theapplication herein is for the identification of objects exterior to thevehicle, many of the same technologies, principles and techniques areapplicable. For example, methods that make use of edge detection areusually central to most neural network-based object classifiers.

An example of such a pattern recognition system using neural networksusing sonar is discussed in two papers by Gorman, R. P. and Sejnowski,T. J. “Analysis of Hidden Units in a Layered Network Trained to ClassifySonar Targets”, Neural Networks, Vol. 1. pp 75-89, 1988, and “LearnedClassification of Sonar Targets Using a Massively Parallel Network”,IEEE Transactions on Acoustics, Speech, and Signal Processing, Vol. 36,No. 7, July 1988.

Pattern recognition techniques as applied to pedestrian recognition arediscussed below.

Another method of identifying a moving vehicle is through the use oflaser vibrometry. This can be done by impinging a laser beam on anobject and analyzing the reflected light using frequency analysis by anymethod such as using a Fourier transform. Different vehicles havedifferent characteristic frequencies.

Another area where pattern recognition is useful relates to determininginformation about a particular geographic location. U.S. Pat. No.6,604,049 discusses a system that is cumbersome and impractical forapplication on normal as opposed to special purpose vehicles. The '049patent solves the problem where an operator wishes to obtain informationabout a building within its vicinity. The operator takes a picture andsends that picture along with the operator's GPS coordinates and viewingdirection to a central server and then requests information about thebuilding. The central server attempts to match the sent image withimages that it has stored in its database to determine what building theoperator is looking at. Of course to cover all buildings and all otherpoints of interest in the United States would require a massive databasethat would require billions of dollars to obtain and maintain renderingthis approach impractical. On the other hand, only one building canoccupy a particular physical location on the earth and thus, if insteadof sending an image the operator sent the GPS-based location of thebuilding, or other point on interest, the required database would beorders of magnitude smaller and manageable on a city server, forexample.

To implement this feature, an observer would measure the location of thepoint of interest relative to the vehicle (or other device for a walkingoperator, for example) and send the coordinates of the point of interestover an appropriate network such as the internet to the appropriateserver along with a request for particular or general information. Thisapproach has many other advantages. A picture of a point of interest canchange based on lighting conditions, time of the year, etc. The locationof the object will not in general change and if it does, the serverdatabase can be updated accordingly. In many cases, the vehicle residentdatabase can have the identification information and additionalinformation may not be needed or if it is the character of thatadditional information needed can be accurately specified and thereforeaccurately answered. For example, what time does that museum close? Sucha system can be used by tourists as well as any other professionalsdesiring information of any nature relative to a particular location.

4. Anticipatory Sensors

FIG. 15 is an angular perspective overhead view of a vehicle 50 about tobe impacted in the side by an approaching vehicle 51, where vehicle 50is equipped with a blind spot monitor or anticipatory sensor systemshowing a transmitter 52 transmitting electromagnetic, such as infrared,waves toward vehicle 51. This is one example of many of the uses of thisinvention for exterior monitoring.

The transmitter 52 is connected to an electronic module 56. Module 56contains circuitry 57 to drive transmitter 52 and circuitry 58 toprocess the returned signals from receivers 53 and 54. Circuitry 58contains a neural computer 59 or a microprocessor with a patternrecognition algorithm or equivalent, which performs the patternrecognition determination based on signals from receivers 53 and 54(FIG. 15A). Receivers 53 and 54 are mounted onto the B-Pillar of thevehicle and are covered with a protective transparent cover. Analternate mounting location is shown as 55 which is in the door windowtrim panel where the rear view mirror (not shown) is frequentlyattached. This dual monitoring of an external space is an example ofstereo vision.

One additional advantage of this system is the ability of infrared topenetrate fog and snow better than visible light, which makes thistechnology particularly applicable for blind spot detection andanticipatory sensing applications. Although it is well known thatinfrared can be significantly attenuated by both fog and snow, it isless so than visual light depending on the frequency selected. (See forexample L. A. Klein, Millimeter-Wave and Infrared Multisensor Design andSignal Processing, Artech House, Inc, Boston 1997, ISBN 0-89006-764-3).Additionally, much of the reflection from fog, smoke, rain or snow canbe filtered out using range gating, which masks reflections that comefrom certain distance ranges from the sensor as discussed above. Also,if the eye-safe part of the spectrum is selected, particularlyfrequencies can be selected to minimize the effect of the sun'sradiation, In the eye-safe region, much higher power can be used andcoupled with range gating, the visibility through fog etc. can besubstantially increased.

Radar systems, which may not be acceptable for use in the interior ofthe vehicle, are now commonly used in sensing applications exterior tothe vehicle, police radar being one well-known example. Miniature radarsystems are now available which are inexpensive and fit within theavailable space. Such systems are disclosed in the McEwan patentsdescribed above. One particularly advantageous mode of practicing theinvention for these cases, therefore, is to use a pulsed or CW radar orpulsed laser radar system, along with a CCD array. In this case, theradar is used to determine distance and the CCD for identification.

In a preferred implementation, transmitter 52 is an infrared transmitterand receivers 53, 54 and 55 are CCD transducers that receive thereflected infrared waves from vehicle 51. In the embodiment shown inFIG. 15, an exterior-deployed airbag 60 is shown which deploys in theevent that a side impact is about to occur as described in U.S. Pat. No.6,343,810 and elsewhere herein. Note, this can be considered a stereocamera implementation if only two receivers are used.

In most of the applications above, the assumption has been made thateither a scanning spot, beam or a line of light will be provided. Thisneed not be the case. The light that is emitted to illuminate the objectcan be structured light. Structured light can take many forms startingwith, for example, a rectangular or other macroscopic pattern of lightand dark can be superimposed on the light by passing it through afilter. If a similar pattern is interposed between the reflections andthe camera, a sort of pseudo-interference pattern can result sometimesknown as Moiré patterns. A similar effect can be achieved by polarizingtransmitted light so that different parts of the object that is beingilluminated are illuminated with light of different polarization. Onceagain, by viewing the reflections through a similarly polarized array,information can be obtained as to where the source of light came fromwhich is illuminating a particular object. Different modulation schemescan also be used to create different patterns and the modulation can bevaried in time for particular applications. These are among the manymethods that can be used to obtain three-dimensional information from atwo-dimensional image. Other techniques based on range gating or camerafocus are described in U.S. patent application Pub. No. 20030209893 andits related predecessors.

As disclosed in U.S. Pat. No. 5,653,462 for interior vehicle monitoringand U.S. Pat. No. 6,343,810 for exterior monitoring, a modulated lightsource can be used to determine the distance to an object eitherinterior or exterior of the vehicle. The basic principle is that thephase of the reflected light is compared to the phase of the transmittedlight and the distance to the reflecting object is determined by thephase difference. There are many ways of implementing this principle.One that has been disclosed is called the photonic mixing device or PMD.In this device, an optical filter is modulated with the same frequencyand the phase that is used to modulate the transmitted light beam. Inthe PMD, this principle is executed on a pixel by pixel basis andincorporated into the CMOS array structure. Although still fallingwithin the teachings of this invention, this results in an unnecessarilycomplicated structure. An alternate method will now be described.

An object in the blind spot or inside a vehicle is illuminated bymodulated light and reflects this light back to a receiver wherein thephase relationship between the reflected light and the transmitted lightis a function of the distance to the reflecting surface. For everypixel, the comparison will be made to the same frequency and phase sinceonly one source of illuminating modulated light has been used toilluminate the entire object. Therefore, there is no advantage inattempting to influence each pixel separately with the modulationfrequency and phase. A similar and preferable approach is to use asingle light valve, electronic shutter or range gate to modulate all ofthe light coming back from the illuminated object.

The technology for modulating a light valve or electronic shutter hasbeen known for many years and is sometimes referred to as a Kerr cell ora Pockel cell. More recent implementations are provided by thetechnology of 3DV discussed above. These devices are capable of beingmodulated at up to 10 billion cycles per second. For determining thedistance to a vehicle in the blind spot, modulations between 5 and 100MHz are needed. The higher the modulation frequency, the more accuratethe distance to the object can be determined. However, if more than onewavelength, or better one-quarter wavelength, exists between the hostvehicle and the object, then ambiguities result. On the other hand, oncea longer wavelength has ascertained the approximate location of thevehicle, then more accurate determinations can be made by increasing themodulation frequency since the ambiguity will now have been removed.

In one preferred embodiment of this invention, therefore, an infraredfloodlight, which can be from a high power laser diode and can operatein the eye-safe part of the spectrum, is modulated at a frequencybetween 5 and 100 MHz and the returning light passes through a lightvalve such that amount of light that impinges on the CMOS array pixelsis determined by a phase difference between the light valve and thereflected light. By modulating a light valve for one frame and leavingthe light valve transparent for a subsequent frame, the range to everypoint in the camera field of view can be determined based on therelative brightness of the corresponding pixels. Pulse or noise orpseudo noise modulation can also be used which has the advantage thatthe return signal can be more easily differentiated from transmissionsfrom other vehicles. Other differentiation schemes are based onsynchronizing the transmissions to the vehicle GPS location, directionof travel or some other such scheme.

Once the range to all of the pixels in the camera view has beendetermined, range gating becomes a simple mathematical exercise andpermits objects in the image to be easily separated for featureextraction processing. In this manner, many objects in the blind spotcan be separated and identified independently.

As mentioned above, it is frequently not possible to separate light froma broad illumination source from reflected sunlight, for example. It hasbeen determined, however, that even in the presence of bright sunlight areflection from a narrow beam of infrared laser light, or a broader beamfrom a high power laser diode, can be observed providing a narrow notchfrequency filter is used on the light entering the receiver. Theprinciples described above, however, are still applicable since asampling of pixels that have been significantly illuminated by thenarrow laser beam can be observed and used for ranging.

The technique of using a wide-angle infrared floodlight is particularlyuseful at night when objects, especially those without self-containedlights, are difficult to observe. During bright sunlight, there isconsiderable information from the visual view taken by the cameras toperform feature extraction, identification, ranging etc. utilizing othertechniques such as relative motion. Thus, a superior blind spotmonitoring system will make use of different techniques depending on theenvironmental conditions.

In more sophisticated implementations of the present invention, therecan be a complementary interaction between the imaging system and theaiming direction of the infrared laser beam. For example, a particularlimited area of the image can be scanned by the infrared system when theimaging system is having difficulty separating one object from another.It is expected, as the various technologies described above evolve, thatvery smart blind spot, anticipatory sensors and general exteriormonitoring systems based on the teachings of this invention will alsoevolve.

In one implementation, the goal is to determine the direction that aparticular ray of light had when it was transmitted from the source.Then, by knowing which pixels were illuminated by the reflected lightray along with the geometry of the transducer mountings, the distance tothe point of reflection off of the object can be determined. Thisrequires that the light source not be collocated with the CCD or CMOSarray. If a particular light ray, for example, illuminates an objectsurface that is near to the source, then the reflection off of thatsurface will illuminate a pixel at a particular point on the CCD or CMOSarray. If the reflection of the same ray however occurs from a moredistant surface, then a different pixel will be illuminated in the CCDarray. In this manner, the distance from the surface of the object tothe CCD can be determined by triangulation formulas.

Similarly, if a given pixel is illuminated in the CCD from a reflectionof a particular ray of light from the transmitter, and the direction inwhich that ray of light was sent from the transmitter is known, then thedistance to the object at the point of reflection can be determined. Ifeach ray of light is individually recognizable and therefore can becorrelated to the angle at which it was transmitted, then the object canbe easily segmented from the background and other objects and a fullthree-dimensional image can be obtained of the object that simplifiesthe identification problem.

The coding of the light rays coming from the transmitter can beaccomplished in many ways. One method is to polarize the light bypassing the light through a filter whereby the polarization is acombination of the amount and angle of the polarization. This gives twodimensions that can therefore be used to fix the angle that the lightwas sent. Another method is to superimpose an analog or digital signalonto the light that could be done, for example, by using an addressablelight valve, such as a liquid crystal filter, electrochromic filter, or,preferably, a garnet crystal array. Each pixel in this array would becoded such that it could be identified at the CCD. Alternately, thetransmitted radiation can be AM or FM modulated to also provide sourceidentification.

The technique described above is dependent upon either changing thepolarization or using the time or frequency domain, or a combinationthereof, to identify particular transmission angles with particularreflections. Spatial patterns can also be imposed on the transmittedlight that generally goes under the heading of structured light, asdiscussed above. The concept is that if a pattern is identifiable, theneither the distance can be determined by the displacement of the patternin the field of view if the light source is laterally displaced from thereceiver or, if the transmission source is located on the same axis butaxially displaced with the receiver, then the pattern expands at adifferent rate as it travels toward the object and then, by determiningthe size of the received pattern, the distance to the object can bedetermined. In some cases, Moiré pattern techniques are utilized. Onemethod of augmenting this in line effect is to have the light passthough a lens that places the focal point of the light in front of theimager, for example, so that it is equivalent to having the light sourceat the focal point. Many other methods of varying the focal point of theillumination source relative to the focal point of the imager are alsopossible such as varying the focal length of the imager and illuminationsource, or having two imagers with different focal lengths.

A further consideration to this invention is to use the motion of theobject, as determined from successive differential arrays, for example,to help identify that there is in fact an object in the blind spot.Differential motion can be used to separate various objects in the fieldof view and absolute motion can be used to eliminate the background, ifdesired.

In another preferred implementation, transmitter 52 is an ultrasonictransmitter operating at a frequency of approximately 40 kHz, althoughother frequencies could be used. Similarly, receivers 53 and 54 areultrasonic receivers or transducers and receive the reflected ultrasonicwaves from vehicle 51.

A “trained” pattern recognition system as used herein is a patternrecognition system that is trained on data representing differentoperating possibilities. For example, the training data may constitute anumber of sets of a signal from receiver 53 represented the returnedwaves received thereby, a signal from receiver 54 representing thereturned waves received thereby and one or more properties of theapproaching object, e.g., its form or shape, size or weight, identity,velocity, breadth and relative distance. Once trained, the trainedpattern recognition system will be provided with the signals fromreceivers 53, 54 and categorize the signals that would lead to adetermination by the system of the property or properties of theapproaching object, e.g., its size or identity.

Some examples of anticipatory sensing technologies follow:

In a passive infrared system, a detector receives infrared radiationfrom an object in its field of view, in this case the approaching objectis most likely another vehicle, and processes the received infraredradiation radiating from the vehicle's engine compartment. Theanticipatory sensor system then processes the received radiation patternto determine the class of vehicle, and, along with velocity informationfrom another source, makes an assessment of the probable severity of thepending accident and determines if deployment of an airbag is required.This technology can provide input data to a pattern recognition systembut it has limitations related to temperature. The sensing of anon-vehicle object such as a tree, for example, poses a particularproblem. The technology may also fail to detect a vehicle that has justbeen started especially if the ambient temperature is high. It also hasdifficulty with the case of a host vehicle sliding into a parkedvehicle, for example. Nevertheless, for use in the identification ofapproaching vehicles the technology can provide important informationespecially if it is used to confirm the results from another sensorsystem.

In a passive audio system one or more directional microphones can beaimed from the rear of the vehicle can determine from tire-producedaudio signals, for example, that a vehicle is approaching and mightimpact the target vehicle, or host vehicle, which contains the system.The target vehicle's tires as well as those to the side of the targetvehicle will also produce sounds which need to be cancelled out of thesound from the directional microphones using well-known noisecancellation techniques. By monitoring the intensity of the sound incomparison with the intensity of the sound from the target vehicle's owntires, a determination of the approximate distance between the twovehicles can be made. This process is aided when a correlation isperformed between the sound of the host tires as modified by the road(cracks, bumps, etc.) and the sound of a vehicle approaching from therear or front on the same lane. Finally, a measurement of the rate ofchange in sound intensity can be used to estimate the time to collision.This information can then be used to pre-position the headrest, forexample, or other restraint device to prepare the occupants of thetarget vehicle for the rear end impact and thus reduce the injuriestherefrom. A similar system can be used to forecast impacts from otherdirections. In some cases, the microphones will need to be protected ina manner so as to reduce noise from the wind such as with a foamprotection layer. This system provides a very inexpensive anticipatorycrash system.

The sensing of sound around a vehicle can also be used as an attentiongetting mechanism to direct other systems to search for the presence ofpotential threats. Thus sensed sound can be used to direct a videocamera, laser spotlight with distance and velocity measuringcapabilities system, radar etc. Naturally, the attention gettingmechanism can also use radar, lasers, ultrasound or other mechanisms.

In a laser optical system design, the transmitter 52 comprises aninfrared laser beam which is used to momentarily illuminate an object asillustrated in FIG. 15 where transmitter 52 is such a laser beamtransmitter. In some cases, a charge coupled device (a type of TVcamera), or a CMOS optical sensor array, is used to receive thereflected light and would be used as one or both of the receivers 53 and54. The laser can either be used in a scanning mode, or, through the useof a lens, a cone of light or a large diameter beam can be created whichcovers a large portion of the object. When the light covers asignificant area a high powered laser diode can be used. When such ahigh powered laser diode is used the distance to the closest reflectingobject can be measured and the intensity of the radiation at thatdistance controlled so as to maintain eye safety conditions or aneyesafe wavelength can be used. If the atmospheric conditions are alsoknown so that the dissipation of the transmitted light can be determinedthen added power can be used to compensate for the losses in theatmosphere still maintaining eye safety conditions. Additionally, thebeam can be made to converge at just the rate to keep the illuminationintensity constant at different distances from the source. To implementsome of these concepts, appropriate lens systems may be required. Insome cases the lenses must respond more rapidly then possible withconventional lenses. Solid state acousto-optical based or liquid basedlenses or MEMS mirrors offer the potential to operate at the requiredspeed.

In each case, a pattern recognition system, as defined above, is used toidentify and classify the illuminated object and its constituent parts.The scanning implementation of the laser system has an advantage thatthe displacement of the object can be calculated by triangulation of thedirection of the return light from the transmitted light providing thesensor and transmitter are displaced from one another. This systemprovides the most information about the object and at a rapid data rate.Its main drawback is cost which is above that of ultrasonic or passiveinfrared systems and the attenuation that results in bad weatherconditions such as heavy rain, fog or snow storms. As the cost of laserscomes down, this system will become more competitive. The attenuationproblem is not as severe as might be expected since the primary distanceof concern for anticipatory sensors as described here is usually lessthan three meters and it is unlikely that a vehicle will be operatedwith a visibility of only a few meters. If the laser operates in theinfrared region of the spectrum, the attenuation from fog is less thanif it is operated in the visible part of the spectrum. As mentionedabove, any remaining atmosphere scattering or absorption problems can bealleviated with range gating.

Radar systems have similar properties to the laser system discussedabove with the advantage that there is less attenuation in bad weather.The wavelength of a particular radar system can limit the ability of thepattern recognition system to detect object features smaller than acertain size relative to the radar wavelength. This can have an effectin the ability of the system to identify different objects andparticularly to differentiate between different truck and automobilemodels. It is also more difficult to use radar in a triangulation systemto obtain a surface map of the illuminated object as can be done with aninfrared laser. However, for anticipatory sensing the object of interestis close to the host vehicle and therefore there is substantialinformation from which to create an image for analysis by a patternrecognition system providing a narrow beam radar is used. Radar remainsa high price option at this time but prices are dropping.

The portion of the electromagnetic spectrum between IR and mm wave radaris called the Terahertz portion of the spectrum. It has the advantageover radar in that optical methods may be able to be used thus reducingthe cost and the advantage over IR in that it is absorbed or scatteredless by the atmosphere. Systems are now being developed which shouldpermit widespread use of this portion of the spectrum.

A focusing system, such as used on some camera systems, could be used todetermine the position of an approaching vehicle when it is at asignificant distance away but may be too slow to monitor this positionjust prior to a crash unless a liquid or other fast focusing lens isavailable. This is a result of the mechanical motions required tooperate the lens focusing system. By itself, it cannot determine theclass of the approaching object but when used with a charge coupled, orCMOS, device plus infrared illumination for night vision, and anappropriate pattern recognition system, this becomes possible. Systemsbased on focusing theory have been discussed above and in the referencedpatents which permit a crude distance determination from two camerasettings that can be preset. In some cases two imagers can be used forthis purpose. A stereo camera-based system is another method of gettingthe distance to the object of interest as discussed above and in thereferenced patents and patent applications.

From the above discussion, it can be seen that the addition ofsophisticated pattern recognition techniques to any of the standardillumination and/or reception technologies for use in a motor vehiclepermits the development of an anticipatory sensor system which canidentify and classify an object prior to the actual impact with thevehicle, and thus provide an output of one of a number of pre-determinedidentities of the object from which waves have been received. Such asystem can be used to deploy external airbags, external nets and/orinterior occupant restraint systems prior to and/or during the accident.Even with a simple imager system using visible or infrared illumination,or no artificial illumination, an estimate of the distance to a vehiclecan be made from a two-dimensional image since the width and sometimesthe height of an identified vehicle is known and its distance can bedetermined by scaling based on its size in the image. This of coursedoes not require the assumption that the road is flat.

FIG. 16 is an exemplary flow diagram of one embodiment of thisinvention. The blind spot monitor begins by acquiring an image of theblind spot that contains an object to be identified (step 159) and bydetermining the range to the object (step 160) and outputs rangeinformation and image information to a feature extraction routine (step161). The output from the feature extraction routine is fed into theneural network or other pattern recognition algorithm. The algorithmdetermines the identity of object (step 162). Once the identity andrange of the object is known then the display can be updated (step 163).Using current and recent information, the relative velocity algorithmdetermines the relative velocity of the object to the host vehicle bydifferencing or by Doppler or other techniques (step 164). With theposition, velocity and identity of the object in the blind spot known,an appropriate algorithm determines whether it is safe for alane-changing maneuver (step 165). If the determination is yes, thencontrol is returned to the image collection and ranging activities and anew image and range is determined. If the lane change determination isno, then a determination is made if the turn signal is activated (whichwould be indicative of the driver's intention to change lanes) (step166). If yes, then audio and/or visual warnings are activated (step167). If no, then a determination is made if the operator has begun tochange the direction of the vehicle to begin executing a lane change(and simply failed to activate the turn signal) (step 168). If map datais present road curvature can also be taken into account. If the vehiclehas begun executing a lane change, then the audio and/or visual warningsare again activated (step 167) and a haptic system begins to exert atorque on the steering wheel to oppose the turning motion of the driver(step 169). Alternately, a vibration can be induced into the steeringwheel, or audio sound, as a further warning to the operator not toexecute a lane change. Following these activities, control is returnedto the image acquisition and range determination activities and theprocess repeats.

The application of anticipatory sensors to frontal impact protectionsystems is shown in FIG. 17 which is an overhead view of a vehicle 70about to be impacted in the front by an approaching vehicle 71. In asimilar manner as in FIG. 15, a transmitter 72 transmits waves 73 towardvehicle 71. These waves are reflected off of vehicle 71 and received byreceiving transducers 74 and 75 positioned on either side of transmitter72.

FIG. 18A illustrates the front of an automobile 76 and shows preferredlocations for transmitting transducer 72 and receiving transducers 74and 75, i.e., the transmitter 72 below the grill and the receivers 74,75on each side of the grill. FIG. 18A also illustrates the distinctivefeatures of the vehicle which cause a distinct pattern of reflectedwaves which will differ from that of a truck 77, for example, as shownin FIG. 18B. In some pattern recognition technologies, the researchermust determine the distinctive features of each object to be recognizedand form rules that permit the system to recognize one object fromanother of a different class. An alternative method is to use artificialneural network technology wherein the identification system is trainedto recognize different classes of objects. In this case, a trainingsession is conducted where the network is presented with a variety ofobjects and told to which class each object belongs. The network thenlearns from the training session and, providing a sufficient number anddiversity of training examples are available, the network is able tocategorize other objects which have some differences from those makingup the training set of objects. The system is quite robust in that itcan still recognize objects as belonging to a particular class even whenthere are significant differences between the object to be recognizedand the objects on which the system was trained.

Once a neural network, or combination neural network, has beensufficiently trained, it is possible to analyze the network anddetermine the “rules” which the network evolved. These rules can thensometimes be simplified or generalized and programmed as a fuzzy logicor neural-fuzzy algorithm. Alternately, a neural computer can beprogrammed and the system implemented on a semiconductor chip asavailable from Motorola.

A goal of an anticipatory frontal crash sensor system is to accuratelypredict the frontal crash severity when an auto accident is about tohappen. The system can be thought of as consisting of two parts: Part 1can use a scanning IR laser, or laser floodlight or spotlight, to detectthe presence of any object within the range of interest and to measurethe distance and speed of the object; and Part 2 can use a camera toclassify the oncoming object (such as car, truck, motorcycle,pedestrian, barrier, pole, etc.). Classification may entail determiningwhich one of a pre-determined identity the object most resembles.

The task of object classification can be divided into the followingthree subtasks based on the increasing complexity:

1) Static Case—stationary camera vs. moving targets.

2) Half Dynamic Case—moving camera vs. stationary targets.

3) Fully Dynamic Case—moving camera vs. moving targets.

Efficient algorithms are being designed specifically for handlingdynamic motion sequences where both the camera and the targets aremoving. The algorithms include the following:

-   -   Algorithm for calculating motion vector with sub-pixel        resolution.    -   Algorithm for estimating background motion due to motion of the        camera.    -   Fast algorithm for obtaining motion vector field for the entire        image.    -   Algorithm for image segmentation based on motion information        alone.    -   Algorithm for finding the region of sky.    -   Algorithm for measuring distance of an object with a single        camera.    -   Algorithm for high-speed lane detection.

The anticipatory sensor system preferably should also be able todetermine the distance, approach velocity and trajectory of theimpacting object in addition to the class of objects to which itbelongs. This can be done with acoustic systems since the time requiredfor the acoustic waves to travel to the object and back determines itsdistance based on the speed of sound. With radar and laser systems, thewaves usually need to be modulated, for example, and the phase change ofthe modulation determined in order to determine the distance to theobject as discussed in U.S. Pat. No. 5,653,462. Since the same distancemeasurement techniques are used here as in above-referenced patentapplications, they will not be repeated.

A radar chip is now available that permits the distance determinationbased on the time required for the radar waves to travel to the objectand back. This technology was developed by Amerigon Inc. of Burbank,Calif. and is being considered for other automotive applications such asconstant distance cruise control systems and backing-up warning systems.

FIG. 18A is a plan front view of the front of a car showing theheadlights, radiator grill, bumper, fenders, windshield, roof and hoodand other objects which reflect a particular pattern of waves whetheracoustic or electromagnetic. Similarly, FIG. 18B is a plane frontal viewof the front of a truck showing the headlights, radiator grill, bumper,fenders, windshield, roof and hood illustrating a significantlydifferent pattern. Neural network pattern recognition techniques usingsoftware available from International Scientific Research. of PanamaCity, Panama can be used to positively classify trucks as a differentclass of objects from automobiles and further to classify differenttypes of trucks giving the ability to predict accident severity based ontruck type and therefore likely mass, as well as velocity. Othersoftware tools are also commercially available for creating neuralnetworks and fuzzy logic systems capable of recognizing patterns of thistype.

In FIG. 19, an overhead view of a vehicle 70 about to be impacted in theside by an approaching vehicle 71 in a perpendicular direction isillustrated where infrared radiation 78 is radiating from the front ofthe striking vehicle 71. An infrared receiver 79 arranged on the side ofvehicle 70 receives this radiation for processing as described above.

The anticipatory sensor system described and illustrated herein ismainly used when the pending accident will cause death or serious injuryto the occupant. Since the driver will no longer be able to steer orapply the brakes to the vehicle after deployment of an airbag which issufficiently large to protect him in serious accidents, it is importantthat this large airbag not be deployed in less serious accidents wherethe driver's injuries are not severe. Nevertheless, it is stilldesirable in many cases to provide some airbag protection to the driver.This can be accomplished as shown in FIG. 20 which is a side view withportions cutaway and removed of a dual inflator airbag system, showngenerally as 80, with an airbag 69 which in essence comprises twoseparate airbags 81 and 82 with one airbag 81 lying inside the otherairbag 82. An optional variable outflow port or vent 85 is provided inconnection with airbag 520 in a manner known in the art. Although asingle inflator having a variable inflation rate capability can be used.FIG. 20 illustrates the system using two discrete inflators 83 and 84which may be triggered independently or together to thereby provide avariable inflation rate of the airbag 69. Inflator 84 and associatedairbag 82 are controlled by the anticipatory sensor system describedherein and the inflator 83 and associated airbag 81 could also beinitiated by the same system. In a less severe accident, inflator 83 canbe initiated also by the anticipatory sensor without initiating inflator84 or, alternately, inflator 83 could be initiated by another sensorsystem such as described U.S. Pat. No. 5,231,253. Each inflator 83, 84contains standard materials therefor, e.g., an initiator, a gaspropellant.

Thus, the variable inflation rate inflator system for inflating theairbag 69 comprises inflators 83, 84 for producing a gas and directingthe gas into the airbag 69, and crash sensors (as described in any ofthe embodiments herein or otherwise available) for determining that acrash requiring an airbag will occur or is occurring and, upon themaking of such a determination, triggering the inflator(s) 83 and/or 84to produce gas and direct the gas into the airbag 69 to thereby inflatethe same at a variable inflation rate, which depends on whether onlyinflator 83 is triggered, only inflator 84 is triggered or bothinflators 83, 84 are triggered (see FIG. 27).

More particularly, the inflator 84 may be associated with ananticipatory crash sensor to be triggered thereby and the inflator 83may be associated with the anticipatory crash sensor or anotherdifferent sensor, such as one which detects the crash only after it hasoccurred. In this manner, inflator 84 will be triggered prior toinflator 83 and inflator 83, if triggered, will supply an additionalamount of gas into the airbag 69.

Although the description above is based on the use of two inflators, thesame result can be obtained through the use of a single inflator and avariable outflow port or vent 158 from the airbag 69 (additionalinformation about a variable outflow port or vent from the airbag 69 isprovided in U.S. Pat. No. 5,748,473 (FIG. 9)) or a flow control valvecontrolling the flow into the airbag. Alternatively, a variable gas flowinflator as disclosed in U.S. patent application Ser. No. 11/131,623,now U.S. Pat. No. 7,481,453, can be used. Such an inflator can use amethod of controlling the inflator burn rate. A schematic drawing of anembodiment including a single inflator and a variable outflow port orvent from the airbag is shown in FIG. 28. This has the advantage thatonly a single inflator is required and the decision as to how much gasto leave in the airbag can be postponed.

As shown in FIG. 28, a first crash sensor 153 is an anticipatory sensorand determines that a crash requiring deployment of the airbag 69 isabout to occur and initiates deployment prior to the crash orsubstantially concurrent with the crash. Thereafter, a second crashsensor 154, which may be an anticipatory crash sensor (possibly even thesame as crash sensor 153) or a different type of crash sensor, e.g., acrush sensor or acceleration based crash sensor, provides informationabout the crash before it occurs or during its occurrence and controlsvent control mechanism 157 to adjust the pressure in the airbag. Thevent control mechanism 157 may be a valve and control system thereforwhich is situated or associated with a conduit connected to the outflowport or vent 85 at one end and at an opposite end to any location wherethe pressure is lower than in the airbag whereby opening of the valvecauses flow of gas from the airbag through the conduit and valve.

Specifically, the vent control mechanism 157 adjusts the flow of gasthrough the port or vent 85 in the airbag 69 (FIG. 20) to enable removalof a controlled amount of gas from the airbag 69 and/or enable acontrolled flow of gas from the airbag 69. In this manner, the airbag 69can be inflated with the maximum pressure prior to or substantiallyconcurrent with the crash and thereafter, once the actual crash occursand additional, possibly better, information is known about the severityof the crash, the pressure in the airbag is lowered to be optimal forthe particular crash (and optimally in consideration of the position ofthe occupant at that moment).

In the alternative, the vent control mechanism 157 can be controlled toenable removal of gas from the airbag 69 concurrent with the generationof gas by the inflator 84 (and optionally 83). In this manner, the rateat which gas accumulates in the airbag 69 is controllable since gas isbeing generated by inflator 84 (and optionally inflator 83, dependent onthe anticipated severity of the crash) and removed in a controlledmanner via the outflow port or vent 85.

4.1 Positioning Airbags

Referring again to FIG. 20, when the large airbag 82 is inflated fromthe driver's door, for example, it will attempt to displace the occupantaway from the vehicle door. If the seatbelt attachment points do notalso move, the occupant will be prevented from moving by the seatbeltand some method is required to introduce slack into the seatbelt orotherwise permit him to move. Such a system is shown in FIG. 21 which isa perspective view of a seatbelt mechanism where a device releases acontrolled amount of slack into seatbelt allowing an occupant to bedisplaced.

The seatbelt spool mechanism incorporating the slack inducer is showngenerally as 68 in FIG. 21 and includes a seatbelt 86 only a portion ofwhich is shown, a housing 87 for the spool mechanism, a spool 88containing several wound layers of seatbelt material 86. Also attachedto the spool 88 is a sheave 89 upon which a cable 90 can be wound. Cable90 can be connected to a piston 92 of an actuator 91. Piston 92 ispositioned within a cylinder 94 of the actuator 91 so that a space isdefined between a bottom of the cylinder 94 and the piston 92 and isable to move within cylinder 94 as described below.

When the anticipatory sensor system decides to deploy the airbag, it canalso send a signal to the seatbelt slack inducer system of FIG. 21. Thissignal is in the form of an electric current which passes through a wire96 and is of sufficient magnitude to initiate a pressure generatingmechanism for generating a pressure in the space between the piston 92and the cylinder 94 to force the piston 92 in a direction to cause thesheave 89 to rotate and thus the spool 88 to rotate and unwind theseatbelt therefrom. More specifically, the electric current through wire96 can ignite a squib 97 arranged in connection with a propellanthousing 95. Squib 97 in turn ignites propellant 98 situated withinhousing 95. Propellant 98 burns and produces gas which pressurizeschamber 99 defined in housing 95, which is in fluid communication withthe space at a bottom 93 of the cylinder 94 between the cylinder 94 andthe piston 92, and pressurizes cylinder 94 below piston 92. Whensubjected to this pressure, piston 92 is propelled upward withincylinder 94, pulling cable 90 and causing sheave 89 to rotate in thecounterclockwise direction as shown in FIG. 21. This rotation causes thespool 88 to also rotate causing the belt 86 to spool off of spool 88 andthereby inducing a controlled amount of slack into the belt and thuspermitting the occupant to be displaced to the side.

In some cases, it may not be necessary to control the amount of slackintroduced into the seatbelt and a simpler mechanism which releases theseatbelt or prevents it from locking up can be used.

An alternate system is shown in FIG. 22 which is a frontal view of anoccupant 100 being restrained by a seatbelt 101 having two anchoragepoints 102 and 103 on the right side of the driver where the one 102holding the belt at a point closest to the occupant 100 is releasedallowing the occupant 100 to be laterally displaced to the left in thefigure during the crash. A detail of the release mechanism 102 takenwithin the circle 22A is shown in FIG. 22A.

The mechanism shown generally as 102 comprises an attachment bolt 111for attaching the mechanism to the vehicle tunnel sheet-metal 109. Bolt111 also retains a metal strip 110 connected to member 106. Member 106is in turn attached to member 108 by means of explosive bolt assembly105. Member 108 retains the seatbelt 101 by virtue of pin 107 (FIG.22B). A stop 112 placed on belt 101 prevents the belt from passingthrough the space between pin 107 and member 108 in the event that theprimary anchorage point 103 fails. Upon sensing a side impact crash, asignal is sent through a wire 104 which ignites explosive bolt 105releasing member 106 from member 108 and thereby inducing a controlledamount of slack into the seatbelt.

In some implementations, the vehicle seat is so designed that in a sideimpact, it can be displaced or rotated so that both the seat andoccupant are moved away from the door. In this case, if the seatbelt isattached to the seat, there is no need to induce slack into the belt asshown in FIG. 23. FIG. 23 is a frontal view of an occupant 115 beingrestrained by a seatbelt 116 integral with seat 117 so that when seat117 moves during a crash with the occupant 115, the seatbelt 116 andassociated attachments 118, 119, 120 and 121 also move with the seatallowing the occupant 115 to be laterally displaced during the crash.

Various airbag systems have been proposed for protecting occupants inside impacts. Some of these systems are mounted within the vehicle seatand consist of a plurality of airbag modules when both the head andtorso need to be protected. An illustration of the use of this module isshown in FIGS. 24A and 24B, which is a frontal view of an occupant 122being restrained by a seatbelt 123 and a linear airbag module 124, ofthe type described in U.S. Pat. No. 6,905,135, including among otherthings a housing 126 and an inflatable airbag 125 arranged therein andassociated inflator. This linear module is mounted by appropriatemounting devices to the side of seat back 127 to protect the entireoccupant 122 from his pelvis to his head. An anticipatory sensor may beprovided as described above, i.e., one which detects that a side impactrequiring deployment of the airbag is required based on data obtainedprior to the crash and initiates inflation of the airbag by the inflatorin the event a side impact requiring deployment of the airbag isdetected prior to the start of the impact. Airbag module 124 may extendsubstantially along a vertical length of the seat back 127 as shown, andthe airbag 124 in the housing 126 may be attached to the seat-back 127or integral therewith. A view of the system of FIG. 24A showing theairbag 125 in the inflated condition is shown in FIG. 24B.

In FIG. 25A, a frontal view of an occupant 128 being restrained by aseatbelt 129 and wherein the seat 131 is displaced toward vehiclecenter, i.e., away from the side and side door of the vehicle, bydeploying airbag 130 is shown. In this case, the seatbelt 129 isattached to the seat 131 as described above with reference to FIG. 23.In this case, rail mechanisms 132 and 133 permit the seat to bedisplaced away from the door under the force produced by the deployingairbag 130. Rail mechanisms 132,133 may include a first member having aguide channel and a second member having a projection positioned formovement in the guide channel of the first member.

To enable displacement of the seat 131 and thus the occupant 128 awayfrom the airbag-deploying structure, the door in the illustratedembodiment, by the force exerted on the seat 131 upon inflation of theairbag 130, the rail mechanisms 132,133 are preferably oriented in anydirection not perpendicular to the deploying direction of the airbag,i.e., not parallel to the side of the vehicle in the illustratedexample. Otherwise, if the orientation of the rails mechanisms 132,133was parallel to the side of the vehicle and the airbag 130 deployed in adirection perpendicular to the side of the vehicle, the seat 131 wouldnot be moved away from the side door. Obviously, to provide for thefastest possible displacement away from the airbag-deploying structure,the rail mechanisms 132,133 are oriented perpendicular to theairbag-deploying structure, which may also be parallel to the deployingdirection of the airbag 130.

Thus, for an airbag mounted in the steering wheel or dashboard anddesigned to deploy in a frontal impact, the rail mechanisms 132,133would optimally be oriented in the longitudinal direction of thevehicle. For an airbag mounted in the side as shown in FIG. 25A, therail mechanisms would optimally be oriented in a direction perpendicularto the longitudinal direction of the vehicle.

In FIG. 25B, a frontal view of an occupant 128 being restrained by aseatbelt 129 and wherein the seat 131 is rotated toward vehicle center,i.e., substantially about an axis perpendicular to a horizontal plane ofthe vehicle, by deploying airbag 130 is shown. In this case, theseatbelt 12 is attached to the seat 131 as described above withreference to FIG. 23. In this case, rail mechanisms 134 and mountinglocations 135 permit the seat to be rotated away from the door under theforce produced by the deploying airbag 130. This figure is shown withthe occupant rotated 90 degrees from initial position, this amount ofrotation may be difficult for all vehicles. However, some degree ofrotation about the vertical axis is possible in most vehicles withappropriate redesign of the vehicle seat and passenger compartment. Railmechanisms 134 may include a first member having a curved guide channeland a second member having a projection positioned for a curving orrotational movement in the guide channel of the first member.

As shown in FIG. 25B, the seat 131 is rotated in a clockwise directionso that the occupant is facing inward during the rotation. The railmechanism 134 can be designed to rotate the seat 131 counterclockwise aswell as along any rotational path. For example, in a frontal impact, itmight be desirable to rotate the occupant toward the adjacent side doorto enable the occupant to exit the vehicle via the side door and/or beextracted from the vehicle via the side door. Otherwise, if the occupantwere to be rotated inward, the seat back would be interposed between theoccupant and the side door and might hinder egress from the vehicle andextraction of the occupant from the vehicle after the crash.

In an alternate case where there is sufficient space for the occupant'slegs and feet, the seat 131 can be rotated as shown in FIG. 25C, i.e.,substantially about an axis oriented in a longitudinal direction of thevehicle. The rotating mechanism comprises a hinged assembly of twoplates 136 and 137, with plate 136 attached to the vehicle floorpan andplate 137 attached to the vehicle seat 131. The two plates are heldtogether by a suitable clamp 138 which is released by the sensor at thesame time the airbag is deployed. Other means for tilting the seat 131or enabling rotation of the seat 131 about the vehicle yaw axis or rollaxis are also envisioned to be within the scope of the invention.

The displacement of the seat 131 by the force exerted by the airbag uponits inflation or deployment is thus very useful for increasing thedistance between the occupant and the site of the impact, whether it isa side impact, frontal impact or even a rear-impact.

Displacement of the seat 131 could also be useful in rollover situationswhere the occupant could benefit from such a rotation or displacementdepending on the nature of the rollover. Such a system could aid inpreventing the occupant's head or other body part from being partiallyejected out of the passenger compartment. Thus, a rollover sensor, whichcan be one or more gyroscopes and/or accelerometers or even an IMU(inertial measurement unit), could replace the crash sensor for thispurpose. Upon detection of a rollover, an action could be taken toinflate an airbag and enable movement of the seat when the force exertedby the inflation of the airbag is effective on the seat. One or more ofthe seat displacement enabling system could be incorporated into thevehicle so that one or more of these systems can be activated upon thedetection of a rollover, depending on which motion of the seat andoccupant would best benefit the occupant.

Some of the techniques disclosed above may not work well for some oftoday's small vehicles. They are more applicable in vans, sport utilityvehicles, some small trucks and some minivans with some modifications.For these and other vehicles, an externally deployed airbag may be analternate solution or both can be used together.

4.2 Exterior Airbags

Once an anticipatory sensor system is in place, it becomes possible toconsider deployment of an airbag external to the vehicle. Thispossibility has appeared in the automobile safety literature in the pastbut it has not been practical until the impacting object can beidentified and/or an assessment of the probable severity of the accidentmade. For prior art systems, it has not been possible to differentiatebetween a plastic sand-filled construction barrier or a cardboard box,for example, neither of which would result in a serious accident (andthus airbag deployment would not be required) and a concrete pillar,tree or wall which would likely result in a serious accident (and thusairbag deployment would be required). With the development of thepattern recognition systems described herein, and in the abovereferenced patents and patent applications, this problem has been solvedand the use of an external airbag now becomes feasible. It is importantto recognize that the effect of an unwanted deployment of an externalairbag is much less severe that for an internal airbag. In the externalcase, the driver should be able to continue to operate the vehicle whichis generally not the case for an internal airbag.

Assessment of the probable severity of the impact is preferablyaccomplished using one or more of the pattern recognition techniquesdisclosed herein, whereby the identity, size or another property of theobject about to impact the vehicle (or with which the vehicle is aboutto impact) is determined using the pattern recognition technique and theidentification or determination of the object's size is consideredbefore initiating deployment of the airbag. In this manner, uponappropriate training of the pattern recognition algorithm, if thevehicle is about to strike a large cardboard box, it will be identifiedas such and airbag deployment will not occur. On the other hand, if thevehicle is about to strike a large truck, it will be identified as suchand airbag deployment will occur. In the prior art, no suchdifferentiation was made about the object involved in the impact basedon remote sensing, i.e., sensing prior to impact.

Such a system adapted for side impact protection is shown in FIG. 26Awhich is a perspective view with portions cutaway and removed of avehicle 140 about to be impacted in the side by another vehicle 141. Anairbag module is shown generally as 146 mounted to the side door of thevehicle 140 prior to inflation of an airbag 147 arranged in the airbagmodule 146. A portion of the side door of vehicle 140 has been cutawayto permit viewing of the airbag module 146. The vehicle 140 contains astrong support beam 144 arranged in a longitudinal direction of thevehicle at least partially within the side door(s) 142 and whichprovides a reaction surface along with the vehicle door 142 for theairbag. Upon initiation by the anticipatory sensor, a deployment door,not shown, is opened in an external door panel 143 by any of a number ofmethods such as pyrotechnically, permitting the airbag 147 to emergefrom the vehicle door 142 as shown in FIG. 26B, the airbag 147 beinginflated by an inflator responsive to the detection by the anticipatorysensor that a side impact requiring deployment of the airbag isrequired.

Through a system such as illustrated in FIGS. 26A and 26B, the accidentcan be substantially cushioned prior to engagement between the vehicleand the impacting object. By this technique, an even greater protectioncan be afforded the occupant especially if an internal airbag is alsoused. This has the further advantage that the occupant may not have tobe displaced from behind the steering wheel and thus the risk to causingan accident is greatly reduced. It also may be the only system whichwill work with some of today's small vehicles. The anticipatory sensorsystem could determine whether the impact is one which requiresdeployment of only the external airbag 147 or one which requiresdeployment of both the internal airbag and the external airbag 147.

Although the description of FIGS. 26A and 26B relates to side impactprotection, it is understood that the same concept can be used forfrontal impacts and rear impacts and rollover situations. That is, thelocation of the airbag 147 is not limited to locations along the side ofthe vehicle, nor to the side door.

An anticipatory sensor system can thus be installed all around thevehicle, with multiple externally deployable airbags, whereby in use,when a determination is made that an object is about to impact thevehicle, only the airbag(s) which will inflate between the vehicle andthe object, and which will cushion the impact, is/are inflated.

For example, FIGS. 29A and 29B show an externally deployable airbagmounted at the front of a vehicle so as to provide frontal impactprotection. The airbag may be mounted in a housing or module in and/orproximate the bumper, fender, grille, or other part at the front of thevehicle. By using anticipatory sensing and/or exterior objectidentification as discussed above, the airbag is deployed prior to or atthe moment of impact.

FIGS. 30A and 30B show an externally deployable airbag mounted at therear of a vehicle so as to provide rear impact protection. The airbagmay be mounted in a housing or module in and/or proximate the bumper oranother part at the rear of the vehicle. By using anticipatory sensingand/or exterior object identification as discussed above, the airbag isdeployed prior to or at the moment of impact.

4.3 Pedestrian Protection

FIGS. 31A and 31B show an externally deployable airbag mounted at thefront of a vehicle for a situation where pedestrian protection isobtained. The airbag 148 may be mounted in a housing or module 150 inand/or proximate the bumper, fender, grille, or other part at the frontof the vehicle. By using anticipatory sensing and/or exterior objectidentification as discussed above, the airbag 148 is deployed prior toor at the moment of impact to protect the pedestrian 152. It can be seenby comparing FIG. 29B and FIG. 31B that the airbag for pedestrianprotection deploys over the hood of the vehicle instead of in front ofthe vehicle. Appropriate positioning and dimensioning of the airbag 148may allow the trajectory of the struck object, i.e., the pedestrian 152,to be controlled. In a similar manner, an airbag for pedestrian impactprotection at the rear of a vehicle would (when the vehicle is backingup) be arranged to deploy over the trunk instead of rearward as shown inFIG. 30B.

In this embodiment, the anticipatory sensor system can be designed tosense an approaching pedestrian or animal and deploy the airbag tocushion the pedestrian's or animal's impact against the vehicle.

In other designs reported in the literature, the hood is raised and/oran airbag is deployed to cushion the impact of a pedestrian with thewindshield. In none of these systems is an anticipatory sensor used andthus they are of marginal value especially for higher speed pedestrianimpacts. The sensors used are relatively crude contact sensors that, issome cases, attempt to distinguish the legs of a pedestrian from a rigidpole, for example (see WO02098715). An alternate approach uses anultrasound in a tube design that stretches along the front bumper and/ora fender of the vehicle which is fully resettable and when theultrasound signal attenuation pulse is used along with a knowledge ofthe vehicle velocity, the pulse can be normalized based on the vehiclevelocity and analyzed by a variety of deterministic or patternrecognition methods such as a neural network. Other sensor technologiesare also applicable including a time domain reflectometer and a fiberoptic bundle of transmitters and receivers randomly terminating and in afoam matrix such that as the bundle is compressed, the transmitters andreceivers get closer together and more light is coupled from the sometransmitters to some receivers. Additionally, capacitive and otherproximity sensors can also be used.

Of particular interest is what is done with the impacted pedestrian.There has been no attention paid to this critical part of the accident.The inventor herein believes that once a pedestrian has been impacted,all measures need to be taken to protect the pedestrian from furtherinjury without causing the driver of the vehicle to lose control of thevehicle. One method would be to use the change of the hood angle andperhaps an airbag to not only cushion the impact of the upper torso andhead of the pedestrian but to also help control his or her trajectory.Additionally, as shown in FIGS. 48 and 49, a net 280 may be deployed,anchored forward in a module or housing 282 in the vehicle (which may bea housing in the same location in which the airbag 148 is housed 150 asshown in FIG. 31A, i.e., along the front of the vehicle), and projectingup so as to catch the pedestrian 152 and hold him or her to the vehiclewhile the vehicle comes to a safe stop (as shown in FIG. 49). The net280 should provide sufficient stretch so to gradually bring thepedestrian up to the vehicle speed while causing a minimum of furtherinjuries. Current practice is to toss the pedestrian into the air andleave him or her to impact with the ground and perhaps be run over byanother vehicle.

The net 280 may be deployed in combination with an airbag 284 as shownin FIGS. 48 and 49 or independent thereof.

When using a net, the pattern recognition system described above whichidentifies the object as being one of a plurality of different,pre-determined identities, e.g., as a human pedestrian, as an animal andas a cyclist, may be further arranged in the anticipatory sensor systemto control deployment of the net based on the identity of the object.For example, deployment of the net to catch or capture the object wouldonly be initiated when the object is a pedestrian or cyclist but wouldnot be deployed when the object is an animal.

Missing from all of the papers and patents which discuss the pedestriandetection problem is any discussion of why a pedestrian should be sensedand if so what are the requirements for such sensing and what will bethe result if a pedestrian is sensed. Based on sensing contact of thepedestrian with some portion of the vehicle, which is clearly too latefor many scenarios, two concepts have been disclosed in patents relatedto pedestrian protection which involve raising hood of the vehicle anddeploying an airbag in the vicinity of the windshield.

The raising of the hood is supposed to reduce the head impact forces andthus injury on the pedestrians head and similarly the airbag should dothe same. Looking at the reduction in distance that the pedestrian'shead would travel between a raised hood and a hood in its normalposition, it is difficult to see how much protection is afforded by thisconcept. A pedestrian probably reaches his maximum rotational velocitysoon after being struck and before his head moves very far.

Similarly, an airbag positioned by the windshield where the pedestrianwould impact after he or she has impacted the hood similarly raisesquestions as to how effective this would be. If the hood were raisedsubstantially so that the pedestrians head struck the hood at a lowervelocity, the driver would not be able to see the road and it couldcause a more serious accident.

An airbag which deployed over the entire hood where the pedestrian islikely to impact would be preferable; however, this would require ananticipatory sensor which has been discussed above. It still might bedesirable to raise the hood somewhat so as to minimize the impact of thepedestrian with the windshield. However this probably could beaccomplished by a properly designed external airbag. The goal of thisairbag would be to cushion the blow of the pedestrian against thevehicle and additionally to affect his or her trajectory, as partiallydiscussed above.

Once the pedestrian has impacted the vehicle and if the vehicle is goingat a substantial velocity, the pedestrian is likely to slide up and overthe vehicle or possibly slide off to one side or the other. In eithercase, the pedestrian is now going to be subjected to a second impactwith the roadway. Thus even if the pedestrian survives the first impacthe or she is likely to be seriously injured by the second impact andpossibly even run over by a following vehicle.

To prevent the second impact, the pedestrian must not leave the strikingvehicle and therefore the pedestrian must be captured and somehowattached to striking vehicle until the vehicle can come to rest. Thus,the hood-located air bag should try to modify the trajectory of thepedestrian to channel him or her so that he or she goes axially with thevehicle. In order to capture the pedestrian at this point some sort ofnet, or similar device or structure, can be deployed to catch thepedestrian and prevent him or her from being subjected to a secondimpact with the roadway. This net should be deployed from as far forwardin the vehicle as possible. It also must be designed so that it getssupport from a position further back in the vehicle to prevent the netcontaining the pedestrian to leave the top of the vehicle and slide downa side.

Let us now look at some timing scenarios. It is estimated that to raisethe hood of a vehicle would require approximately 100 milliseconds. Todeploy an airbag of the type described above would probably take alsoabout 100 milliseconds. To deploy a net similar to that described abovewould probably take in excess of 250 milliseconds. Of course none ofthese actions would be taken unless the system was virtually one hundredpercent sure that the object that the vehicle is about to strike is apedestrian. False positives cannot be tolerated as they could cause thevehicle to have an accident. If the system is designed for a worst casescenario, the timing for making a deploy decision should be based on avelocity such as 60 mph. Other philosophies could be selected provided adetailed study shows that the probability of a 60 mph pedestrian impactis close to zero. Let us start with the 60 mph assumption and see whatthe implications are for timing and accuracy of the system.

Assuming 300 milliseconds are necessary to deploy the net, airbag andhood rise or other appropriate system, a final decision to deploy thesystem would have to be made when an object is approximately 10 metersaway in front of the vehicle for a vehicle traveling approximately 60mph. This provides 50 milliseconds for an image to be acquired andanalyzed. To summarize, the system requirement for a single image systemis that there can be no false positives and as high an accuracy ofdetection as possible for an object 10 meters from the vehicle if atotal of 50 milliseconds is provided for acquiring the image andanalyzing the data. Detection accuracy under these circumstances, asbelieved by the inventor, should be higher than 95% and preferablyhigher than 99%.

Naturally, there is additional information that can be used to improvethe accuracy of the system. However, the single frame accuracyrequirement cannot be dismissed since there are numerous situationswhere a pedestrian can seemingly appear out of nowhere such as when avehicle is making a relatively sharp turn or passing an obstacle to theroad that blinds the driver until the object is quite close to the hostvehicle. Another example is a car which suddenly swerves out of a lanebecause of a pedestrian standing in the lane.

Other information is of course available to the system much of the time.For example if a pedestrian is detected even with low probability ofclassification at one hundred meters from the vehicle, then the vehiclecan begin slowing down. This will permit a larger number of images to beacquired before the pedestrian time-to-contact is less than 300milliseconds. This extra information certainly should be used to improvethe detection accuracy. However, there are many situations where a caris turning and a pedestrian comes in view which temporarily is in thepath of the vehicle but soon is not. This pedestrian protection systemwould become very unpopular if the vehicle began breaking in every suchsituation. Of course if the vehicle had an accurate map onboard and knewprecisely its location, then it would know that as long as it stayed onthe road that it would not impact the pedestrian. Early detection of apedestrian also can give the driver or an appropriate system time tochange the course of the vehicle and, if this is done by a vehiclecontrol system, knowledge of the road geometry and the obstacles in thevicinity of the pedestrian and the vehicle would have to be accuratelyknown.

Two other issues must be considered. The first relates to where thevehicle is being driven. Most of the related art papers assume that thevehicle would be driving in an urban area and thus generally at lowerspeeds where there is a large number of opportunities to impactpedestrians. Under such driving conditions, there should be a lowerimpact velocity at which the hood rising, airbag and/or net would not bedeployed. A five mile per hour impact with a pedestrian can still causeserious injury to the pedestrian but the pedestrian's velocity may notbe sufficient to require deployment of the pedestrian catching net.

In rural settings however, many types of animals can appear in the pathof a vehicle and these must not be confused as being pedestrians.Certainly, no driver would want to catch a large deer in a net on top ofhis or her car. Thus, the imaging system used must be able todistinguish animals from humans. This can be especially difficult for adeer which is facing the vehicle on a road since it can have a similarshape and thermal emission as a human. Pre-crash maneuvering andbreaking would be appropriate in rural settings when there is any objectin the path of the vehicle. Once again however this situation can besignificantly improved if the path of the vehicle is known. This wouldavoid the problem of a deer on the side of curving road.

There are many methods of detecting pedestrians as discussed above andall are contemplated by the inventor herein. Among others these includethe use of radar, terahertz, or any frequencies in the IR portion of thespectrum as well as the visual portion. One preferred solution is to usea near infrared frequency such as SWIR (above 1.4 microns) in the formof a laser spotlight which would pass eye safety requirements asdiscussed above and in the referenced patents. This laser spotlightcoupled with range gating permits the easy segmentation of objects inthe scene and thus the rapid classification using the modular neuralnetwork or combination neural network system available fromInternational Scientific Research.

When the entire scene is being observed or where the range gate is setfor maximum-to-minimum range, then an image containing multiple objectscan occur and the objects to be analyzed first can be segmented byreducing the maximum range gate until a single object is left. Incontrast to the Shashua paper referenced above, a preferred approachherein could be to feed the entire image in the region of interest intoan edge detection algorithm and then into a first neural network forrough classification followed by other neural networks for more preciseclassification. Alternatively, the entire image containing multipleobjects of interest can be fed to an edge detection algorithm and thento a neural network having multiple outputs which would returnapproximate identities of all of the objects in the image into differentoutputs slots. The neural network would output that there is a likelypedestrian centered at pixel 32,128 and a vehicle at pixel 200,100 orsome other such output. The system can then concentrate on the objectwhich would be the most threatening to human life until that object hasbeen positively identified and eliminated as a threat, or all objectscan be treated simultaneously depending on the computational power ofthe system.

It is envisioned that the features of the side impact protectionsystems, rear impact protection systems, frontal impact protectionsystems, and pedestrian impact protection systems can be usedinterchangeably to the extent possible. Thus, features of the sideimpact protection systems can be used for rear, frontal and pedestrianimpact protection.

4.4 Rear Impacts

The impact of a vehicle into another resulting in whiplash injuries,although rarely fatal, is the most expensive accident. This problem andmethods of solving it are discussed in U.S. Pat. No. 6,784,379 and itsrelated applications, all of which are incorporated herein by reference.For anticipatory sensors disclosed herein and used for rear impacts,since the severity of an impact can be predicted, they can initiatedeployment of resettable systems for lower severity accidents and ofnon-resettable systems such as airbags in higher severity accidents.Resettable systems include, for example, seatbelt tensioners andheadrest movement systems such as described in JP2003-112545 and U.S.Pat. No. 5,580,124, and elsewhere.

4.5 Positioning of Out-of-Position Occupants

In another embodiment of the invention using an anticipatory sensorsystem, a deploying airbag is used to position the occupant. That is, anairbag is arranged and designed to move only a part of the occupant, notnecessarily the seat, so as to make room for deployment of anotherairbag. For example, a shoulder or thorax airbag could be deployed basedon a determination from an anticipatory sensor system that a crash isimminent and a determination from an interior monitoring system that theoccupant's head is resting against the window. The deploying shoulder orthorax airbag would serve to push the occupant's head away from thewindow, making room for the deployment of a side curtain airbag betweenthe window and the person's head. Such positioning airbags could bestrategically arranged in the vehicle to move different parts of anoccupant in a specific direction and then deployed based on the positionthe occupant is in prior to the impact to change the occupant's statusof “out-of-position” vis-à-vis airbag deployment to “in-position”.

An example of the use of positioning airbags in accordance with theinvention is shown in FIGS. 32A, 32B and 32C wherein a passenger 275 isshown leaning against a door 276 in FIG. 32A, a positioning airbag 277deploys from the door 276 to move the passenger 275 away from the door276 as shown in FIG. 32B and a side curtain airbag 278 is deployed,e.g., from a location above the window, when the passenger 275 has beenmoved away from the door 276 as shown in FIG. 32C. Ideally, thepassenger 275 or a part thereof would be moved a sufficient distance toenable effective deployment of the side curtain airbag while preventinginjury. Such a positioning airbag can be initiated by an anticipatorysensor as taught herein.

Using a positioning airbag, the positioning airbag is preferablydeployed before the main airbag or side curtain airbag. Deployment ofthe positioning airbag could be initiated based on anticipatory sensingof an object about to impact the vehicle, and/or in conjunction with aninterior monitoring system or occupant position sensor which would sensethe position of the occupant or a part thereof and determine the need tomove the occupant or a part thereof to enable deployment of the mainairbag or side curtain airbag, or at least to make the deployment moreeffective. Deployment of the positioning airbag(s) could also be basedon an actual detection of a crash involving the vehicle by crush-basedsensors or acceleration-based sensors and the like. Determination of theposition of the occupant can lead to assessment of a situation when theoccupant is out-of-position for deployment and must be moved into abetter position for deployment of the main or side curtain airbag. A“better” position for deployment being a position in which the occupantis able to receive more of the benefits of the protective cushion and/ormovement prevention provided by the main or side curtain airbag.Positioning airbags can be especially useful for cases where arelatively slow rollover event is taking place in order to position anoccupant for a side curtain airbag deployment.

The use of positioning airbags is also particularly suited forrollovers. In a rollover situation, a vehicle 170 can move sidewaystoward a curb or highway barrier 171 (FIG. 33A) and then strike thebarrier 171 (FIG. 33B). Upon impact with the barrier 171, the driver 172is forced toward the driver-side window 173 while the vehicle 170 beginsto rollover. At this time, as shown in FIG. 33C, the side curtain airbag174 deploys downward. However, since the driver 172 is against thewindow 173, the side curtain airbag 174 may actually deploy inward ofthe driver 172 thereby can trap the driver 172 between the side curtainairbag 174 and the window 173. Typically the window 173 breaks so thatthe head of the driver 172 may actually be forced through the brokenwindow. When the vehicle 170 completes the rollover, the driver 172 isforced against the ground and may be seriously injured if not killed(FIG. 33D).

To remedy this situation, the invention contemplates the use of one ormore positioning airbags. As such, as shown in FIG. 34B, when the driver172 is detected to be against the window 173 or simply displaced from aposition in which the side curtain airbag 174 will be properly deployed,a positioning airbag 175 is deployed. Such detection of the position ofthe occupant may be made by any type of occupant sensor including butnot limited to a proximity sensor arranged in the door. As shown in FIG.34B, the positioning airbag 175 is arranged in a door of the vehicle 170and deploys inward. The positioning airbag 175 may also be arranged inthe side of the seat or in the side of the vehicle other than in a door.Also, the positioning airbag 175 can be controlled to deploy wheneverdeployment of the side curtain airbag 174 is initiated. That is, it ispossible to provide a sensor for detecting when side curtain airbag 174will deploy (for example a rollover sensor) and once such deployment isauthorized, the positioning airbag 175 will be deployed prior todeployment of the side curtain airbag 174 to ensure that the driver isproperly positioned (See FIGS. 34C and 34D). In this case, theactivation mechanism of the positioning airbag 175 is coupled to thecontrol device of the side curtain airbag 174.

Although a side curtain airbag has been illustrated in the variousfigures herein, this same invention applies when an inflatable tube ortubular airbag, such as manufactured by Simula Inc. of Arizona, is usedfor retention of an occupants head within the vehicle. Thus, for thepurposes herein, a side curtain airbag will encompass such inflatabletube or tubular airbags or equivalent. Similarly, although they have notbeen widely used up until now, other systems employing nets have beenproposed for this purpose and they are also contemplated by thisinvention.

In one typical embodiment, the positioning airbag will always bedeployed in rollover accidents when the curtain or tubular airbag isdeployed and an occupant position sensor is not used. Similarly, theside curtain or tubular airbag is also usually deployed even in sideimpacts where there is no rollover as it provides protection against theoccupant's head striking the intruding vehicle. There is even a strongmotivation for deploying both side positioning and curtain or tubularairbags for frontal impacts as control over the position and motion ofthe occupant is improved.

Finally, although it is desirable to deploy the positioning airbagfirst, in many cases both airbags are deployed at the same time and thefact that the positioning airbag will deploy more rapidly is relied onto prevent the entrapment of the occupant's head outside of the window.The flow of gas into the curtain airbag can be controlled to facilitatethis effect.

In a frontal crash when a frontal protection airbag is used, thedirection of deployment of the positioning airbag would be substantiallyperpendicular to the direction of deployment of the frontal airbag. Thatis, the positioning airbag would deploy in a direction away from thedoor laterally across the vehicle whereas the main airbag would deployedin a longitudinal direction of the vehicle. The positioning airbag wouldthus move the occupant laterally to obtain the benefits of thedeployment of the frontal airbag. However, the angle between thedeployment direction of the positioning airbag and the deploymentdirection of main or side curtain airbag can vary. In fact, it isconceivable that the deployment directions are the same whereby if anoccupant is too close to the deployment door or location of the mainairbag, then a smaller positioning airbag is deployed to push theoccupant away from the deployment door and only once the occupant issufficiently distant from the deployment location is the main airbagdeployed. Monitoring of the position of the occupant is useful todetermine when the positioning airbag need to be deployed and if andwhen the occupant is moved a sufficient distance by the deployment ofthe positioning airbag so as to be positioned in a proper position fordeployment of the main or side curtain airbag. The rate of deployment ofthe positioning airbag and the amount of inflation gas used to deploythe airbag can be varied depending on the size and position of theoccupant (as determined by occupant sensors for example) and theseverity of the crash.

The timing of the deployments of the positioning airbag and main airbag,or the positioning airbag and side curtain airbag, can take into accountthe distance the occupant must be moved, i.e., the position of theoccupant. This can ensure that the occupant is not moved too far by thepositioning airbag out of the range of protection provided by the mainairbag. The timing can thus be based on the position and/or weight ofthe occupant. The timing of the deployments can also or alternatively bebased on the characteristics or properties of the occupant, i.e., themorphology of the occupant. For example, different deployment scenarioscan be used depending on the weight of the occupant since a lighteroccupant would move faster than a heavier occupant.

The rate of deployment of the main or side curtain airbag can also bevaried so that it deploys more slowly than the positioning airbag. Assuch, the positioning airbag will have its positioning effect first andonly thereafter will the main or side curtain airbag have its protectiveeffect. The rate of deployment of the airbags can be varied in additionto the timing of the deployments.

Although the use of positioning airbags is described above withreference to FIGS. 34A-34D for a driver, it is understood that suchpositioning airbag can be used for each seating location in the vehicle.Also, one airbag can be used for multiple seating locations, forexample, seating locations on the same side of the vehicle.

The manner in which the positioning airbag is deployed is illustrated inFIG. 35 wherein the first step is to detect a rollover or other crashsituation at 240. Such detection may be based on anticipatory crashsensing. The occupant's position (present or future) may then bedetermined or monitored at 241. For multiple occupants, the position ofeach occupant can be determined at that time or it is conceivable thatthe occupant's position at a set time in the future is extrapolated, forexample, based on the occupant's current position and velocity, theoccupant's position in the immediate future can be calculated using theequation that the future position equals the current position plus thevelocity times the time differential. A determination is made at 242whether the occupant is “out-of-position” which in the rolloversituation would be too close to the window. If not, then the sidecurtain airbag is deployed at 243. If yes, a positioning airbag isdeployed at 244 to move the occupant into an appropriate position fordeployment of the side curtain airbag. Optionally, the occupant'sposition can be determined after deployment of the positioning airbag at245 or extrapolated based on the imparted velocity to the occupant fromthe deploying positioning airbag. If the position of the occupant isproper for deployment of the side curtain airbag or will be proper forthe deployment of the side curtain airbag, then the side curtain airbagis deployed at 243.

Positioning airbags can be arranged at different locations throughoutthe vehicle with each one designed to move one or more occupants in adesired direction. A control unit for the positioning airbags, which maybe a processor coupled to a crash sensor system (anticipatory or other)and occupant position determining system, determines which positioningairbag(s) are to be deployed based on the position of the occupant(s).

The general components of an apparatus for deploying multiple airbags inaccordance with the invention are shown schematically in FIG. 36. Acrash and/or rollover sensor system 180 is arranged on the vehicle andmay include one or more anticipatory crash sensors, crush crash sensors,acceleration-based crash sensors or rollover sensors based ongyroscope(s), an IMU or angle sensors. An occupant position/velocitysensor system 181 is arranged on the vehicle to monitor the presence andposition of the occupants and optionally determine the velocity of theoccupants. The sensor system 181 can be any known system in the priorart including those disclosed in the assignee's U.S. patents referenceabove. Sensor system 181 can also include sensors which measure amorphological characteristic or property of the occupant such as theoccupant's weight. The crash sensor system 180 and occupant sensorsystem 181 are coupled to a processor/control unit 182. Control unit 182receives input from the crash sensor system 180 as to an expected crashinvolving the vehicle (when the crash sensor system 180 includes ananticipatory sensor) or an actual crash or rollover involving thevehicle. Control unit 182 determines which protective airbags need to bedeployed, if any, to protect the occupants. Such protective airbagsinclude the side curtain airbag on the left side of the vehicle 183, theside curtain airbag on the right side of the vehicle 184, the frontairbag on the left side of the vehicle 185, the front airbag on theright side of the vehicle 186, and others. Control unit 182 alsodetermines whether any of the positioning airbags 1-4 (elements187A-187D) need to be deployed prior to and/or in conjunction with thedeployment of the protective airbags. Although generally the positioningairbags are deployed prior to the deployment of the protective airbagsin order to properly position an occupant, the positioning airbags couldbe deployed whenever the vehicle experiences a crash or rollover toprovide some added cushioning. Positioning airbag 1 could be associatedwith the side curtain airbag on the left side of the vehicle andeffective to move the occupant(s) away from the left side of thevehicle. Positioning airbag 2 could be associated with the side curtainairbag on the right side of the vehicle and effective to move theoccupant(s) away from the right side of the vehicle. Positioning airbags3 and 4 would serve to deploy to position the occupants for deploymentof the respective frontal airbags.

Control unit 182 can determine the timing of the deployment of thepositioning airbag and associated protective airbag, i.e., the timedifferential between the initiation of the inflation which will beoptimum to allow the occupant time to be moved by the positioning airbaginto position to be protected by the protective airbag. Control unit 182can also determine the rate of inflation of the positioning andprotective airbags, when such airbags are provided with the capabilityof variable inflation rates. In this case, the protective airbag may bedeployed at the same time as the positioning airbag (or possibly evenbefore) but the protective airbag inflates more slowly than thepositioning airbag. Control unit 182 can also factor in the morphologyof the occupant to be protected when determining the inflationparameters, i.e., the timing difference and rate of inflation. This isuseful since weight of the occupant affects the occupant's movement,i.e., a heavier occupant will be moved more slowly than a lighteroccupant. In some cases more gas will be allowed to flow into the airbagfor heavier people than for lighter people.

5.0 Agricultural Product Distribution Machines

Referring now to FIGS. 37-46, embodiments of vehicles in accordance withan invention herein including a new ground speed sensor will bedescribed. The ground speed sensor is designed to more accurately detectthe speed of travel of the vehicle on the ground than in prior artconstructions discussed above. Specifically, the ground speed sensor inaccordance with the invention accounts and compensates for pitching ofthe vehicle and/or slipping of the wheels which would provide anincorrect ground speed using prior art constructions.

Generally, the ground speed sensor is mounted on a frame of the vehicleand includes a wave transmitter or emitter and wave receiver. The wavetransmitter or emitter directs waves at an angle from the vertical planeso that the waves impact the ground and reflect from the ground back tothe wave receiver. A processor or other type of computational deviceconsiders the information about the transmission or emission of thewaves and the reception of the waves and calculates the speed of travelof the vehicle on the ground based on the information. The informationmay be the frequency of the transmitted and received waves, the time oftransmission and the time of reception of the waves and other similarparameters. The speed of travel of the vehicle on the ground may bedetermined based on a plurality of time intervals between transmissionof waves and reception of waves. The processor can also be designed todetermine the speed of travel of the vehicle on the ground based on ameasurement of the Doppler frequency.

The ground speed sensor may be used with various agricultural machineryand construction equipment as described below. Once the ground speed isdetermined, the agricultural machinery can be controlled to distributeproduct based in part on the ground speed or the construction machinerycan be operated based in part on the ground speed.

Compensation for pitching of the vehicle may involve mounting collocatedtransmitter and receiver on the frame to move in a curve about a centerof rotation of the vehicle upon pitching of the vehicle. The absolutepitching angle of the vehicle is measured and a reference angle is alsomeasured with the pitching angle of the vehicle relative to the groundbeing the difference between the absolute pitching angle and thereference angle. The reference angle may be measured using a gravity ortilt sensor. The absolute pitching angle of the vehicle may be measuredby a two antenna GPS receiver.

Various constructions of the wave transmitter and receiver may be usedincluding those discussed above. For example, the wave transmitter maycomprise one or more laser diodes, a pulsed laser and a continuous laserbeam directing infrared light to scan in a line. In the latter case, atransmitter control unit controls the scanning laser beam of infraredlight such that the infrared light traverses an area of the ground nearthe vehicle. The wave receiver may comprise a single pixel receptor, aCCD array, a CMOS array, an HDRC camera, a dynamic pixel camera and/oran active pixel camera.

The wave transmitter and receiver can be designed to transmit ultrasonicwaves, radar waves and infrared waves among others. In the latter case,a notch filter may be used for filtering light other than infrared lightemitted by the wave transmitter. A light valve can also be used as thewave receiver.

Referring now to FIG. 37, a known product distribution machine 190generally includes a frame, at least one tank, bin or other storagecompartment 191 for holding product to be distributed, and adistribution system or other conveying mechanism 193 to transmit theproduct to a desired location such as the field. A driving mechanism 192passes the product from the storage compartment 191 to the conveyingmechanism 193. The storage compartment 191, driving mechanism 192 andconveying mechanism 193 may be mounted on the frame or may constitutethe frame.

Referring now to FIGS. 38 and 39, the control unit of the productdistribution machine 190 generally includes a microprocessor 194, whichenables the driving mechanism 192 to run automatically. The agriculturalproduct distribution machine 190 is towed by a vehicle (not shown) in afield onto which product must be applied. In one embodiment of theinvention, the microprocessor 194 receives command signals from a groundspeed sensor 195 and from a user interface 196. The ground speed sensor195 detects the ground speed or forward speed of the tractor, not shown,that is towing the agricultural product distribution machine 190 acrossthe field.

The user interface 196 allows the operator to monitor and set variousparameters relating to the process, such as application rate, locationin the field, implement widths, calibration numbers, and the like. Someof the process parameters can be changed through controls or operatorsettings 197.

The operator settings 197 on the user interface 196 may be buttons orany other input device, such as keys on a keypad, switches, levers, orthe like, mouse pad or even voice input. The user interface 196 ispositioned in such a way that an operator can control the system whilethe agricultural product distribution machine 190 is traveling on thefield. The user interface 196 can comprise a display and a consolesituated in the cab of the tractor towing the agricultural productdistribution machine 190.

The ground speed and the data from the user interface 196 are processedby the microprocessor 194. Upon processing, the microprocessor 194activates the driving mechanism 192 that, in turn, drives product fromthe storage compartment 191 into the conveying mechanism 193 at acontrolled rate output 198.

During operation, the driving mechanism 192 runs automatically at adispensing rate calculated based on the detected ground speed, on theoperator settings, and on other process specific parameters. Therefore,the controlled rate output 198 varies with the rate data input tocompensate for ground speed fluctuations and to produce a consistentapplication of the product onto the field.

The system in accordance with the invention assumes that detailedmeter-by-meter variation in the distribution of the product onto thefield is not required. Instead, the goal is the most even distributionof product as possible. Under these assumptions, normally the farmerwill try to be certain that every part receives the optimum amount ofproduct and thus he will tend to slightly over apply product in order toaccount for the inaccuracy in the ground speed sensor 195 for the casewhere it over calculates the true ground speed. On the other hand, whenthe wheels are slipping, even more product will be dispensed per linearmeter. Thus, in normal operation there will be an uneven distribution ofproduct and it will tend to result in the consumption of an excess ofproduct. Under these assumptions, by practicing the invention, thefarmer will save money as now he will not have to over-dispense producton average and in particular when the wheels are slipping. In the othercase where the farmer truly dispenses the right amount of product onaverage but the distribution is uneven under the prior art system, hewill be penalized with inferior yield and thus lose money at harvesttime compared with what he will receive when practicing this invention.

The invention will be described next in the context of three preferredembodiments: air seeding systems, precision planters, and sprayers. Theperson skilled in the art will recognize that the present invention maybe embodied in other types of product distribution machines.

Air Seeders

FIGS. 39 and 40 depict an air seeder or air cart 200 (which is anagricultural product distribution machine 190) embodying the presentinvention. The air cart 200 includes an air distribution system 203(which is a conveying mechanism 193). The air cart 200 also includes aseeding tool 210, which may be a series of ground openers. The airdistribution system 203 includes a manifold 204, and in someembodiments, a series of hoses. The air cart 200 can be attached to avehicle, such as a tractor, or it can be built as an integral part of avehicle. The air cart 200 includes one or more tanks 202 (whichconstitute a storage compartment) to hold products like seed andfertilizer. The air cart 200 also includes a driving mechanism 192. Thedriving mechanism 192 includes a metering system 207 to deliver theappropriate amount of product to the air distribution system 203, and afan 205.

The metering system 207 controls the dispensing rate of product from thetanks 202 into the air distribution system 203. The dispensing rate ofthe metering system 207 determines the application rate of product ontothe field.

Referring now to FIG. 40, the metering system 207 includes a meteringwheel 208 designed to dispense product at a predetermined rate. Asproduct passes through the metering system 207, it is carried by airflowprovided by the fan 205 through the manifold to headers 206, where theproduct is split into several runs and then passes through the groundopener or seeding tool 201 and into the furrow created by the groundopener or seeding tool 201.

The metering system 207 is driven automatically by a variable rate drivemechanism. In the case of a metering wheel 208, the variable rate drivemechanism will rotate the metering wheel 208 at various rates. Manydesigns of variable rate drive mechanisms are known in the art and canbe used in embodiments of the present invention.

The air cart 200 also comprises sensing equipment, including a groundspeed sensor 212 for detection of the ground speed of the air cart 200.Thus, variations in the ground speed of the air cart 200 may be takeninto account when calculating the application rate, so that seeds can bedispensed evenly. The accuracy with which the seeds are distributeddepends on the accuracy of the ground speed sensor 212 as discussedabove.

FIG. 41 is a block diagram of a system in accordance with the presentinvention as it applies to seeders and planters. The microprocessor 211receives signals from the ground speed sensor 212 and the user interface213, such as the desired rate, the implement width and the calibrationvalue. A feedback loop returns to the microprocessor 211 the rotationrate of the metering wheel 208 at any moment (the RPM from themeter/singulator 214 which is comparable to the metering wheel 208).Based on this information, the microprocessor 211 calculates the desiredrate at any moment and commands the metering wheel 208 to rotate at thedesired rate. As previously indicated, the user interface 213 comprisesoperator setting buttons 197.

Planters

Like the air seeders, planters have several tanks for holding seed orfertilizer, and an air distribution system comprising a series of hoses.Product travels through the hoses, entering through a series of inletsinto several chambers for storing the product. In one embodiment, eachchamber has joined to it a fingered singulator disk. Each chamber islocated just above a corresponding ground opener. The singulator diskrotates such that as each finger passes the place where product puddlesinto the chamber, a single seed/fertilizer falls into the finger. Thedisk continues to rotate such that each subsequent finger can pick upproduct. The filled fingers pass a brush that eliminates the chance ofmultiple seeds being in a single finger. The filled fingers pass anotheropening in the disk when the product is dropped onto an elevator openingthat carries the product to the ground opener.

The driving mechanism 192 of the planters can operate in a mode aspreviously discussed. FIG. 41 applies to planters, as well as, seeders.

The driving mechanism 192 of the planter is activated into rotating thesingulator disk 214 at the controlled rate output 370. In this mode, thecontrolled rate output 198 is a function of the operator settings and ofthe detected ground speed.

Sprayers

Referring now to FIG. 42, a basic sprayer is depicted. Generally, asprayer has at least one storage compartment 221 for chemicals. In anembodiment of the invention, the compartment 221 contains a pre-mixedchemical ready for distribution. In an alternative embodiment, thecompartments 221 store only water and, as the water travels through thedistribution lines 223, the water is injected with the correct amount ofchemical. The required gallons/acre ratio is known and programmed into amicroprocessor 211, connected to the user interface 213 in the cab ofthe tractor towing the sprayer. The gallons/acre ratio is dependent uponthe type of crop, the type of chemical, the position in the field, andthe like. A pump 224 pushes the product into the distribution lines 223.As product is pushed into the distribution lines 223, it travels down atthe flow rate necessary to dispense the required gallons/acre out of thenozzle on the spray bar. During this process, the entire systemautomatically remains pressurized at the appropriate level. The flowrate is dependent upon the ground speed of the sprayer.

In FIG. 43, the invention is shown applied to a vehicle speed detectionsystem incorporated in a tracklaying vehicle such as a bulldozer. Thisvehicle speed detection system comprises a Doppler anddisplacement-measuring sensor 226 mounted on a rear part (e.g., ripperframe) of the vehicle body of a bulldozer 225 at a level h. The Dopplerand displacement-measuring sensor 226 transmits an electromagnetic orultrasonic wave beam 228 toward the ground 227 at a specified beamdepression angle θ and receives a reflected wave from the ground 227.The ground speed of the bulldozer 225 is calculated based on thesetransmitted and received waves. The normal variation in the mountingangle is normally around φ=±5 degrees.

Referring now to FIGS. 44A and 44B which illustrate the logic of thecorrection executed in the vehicle speed detection system to eliminatethe influence of the pitching angle of the vehicle.

FIG. 44A shown the relationship between the horizontal speed (v) of thevehicle 225 and the speed of the vehicle 225 in the direction that thewaves are transmitted and is described by v cos(θ). Therefore, therelationship between Doppler shift frequency Δf and the vehicle speed vis described by the following equation (2), which is basically the sameequation as (1).Δf=2fV cos(θ)/C  (2)

where:

f is transmission frequency, and

C is electromagnetic wave propagation velocity.

In this embodiment, the sensor 226 moves in a curve about the center ofrotation of the vehicle 225 when the vehicle 225 pitches. To this end,the sensor 226 is movably mounted to a frame of the vehicle 225.

The pitching movement rotates the sensor 226 changing its effectivetransmission angle relative to the ground 227. If the angle of thevehicle 225 relative to the ground 227 is known due to pitching, thenthe angle θ can be corrected and the Doppler velocity determined. Thiscan be accomplished using an angular rate sensor as described in U.S.Pat. No. 6,230,107 or by using a tilt sensor such as manufactured byFrederick.

However, an angular rate sensor only measures the change in the pitchangle and therefore it requires that a reference be provided such as bya gravity or tilt sensor, otherwise the results are inaccurate. Evenusing the tilt sensor poses problems as to when the sensor is giving anaccurate reading in the presence of frequent and rapid angular motions.Pitch can also be accurately measured if a two antenna GPS receiver ispresent but then the system begins to get expensive. This is not atotally illogical solution, however, since GPS receivers are rapidlybecoming less expensive as they are put in cell phones to satisfyfederal 911 mandates. Of course, all of these solutions fail if thevehicle is operating on a hill unless an accurate map is available. Whatis desired is the velocity of the vehicle relative to the ground even ifthe ground is sloping.

The present invention solves this problem by measuring both the Dopplervelocity and the distance that the transmitted waves travel to theground 227 and return. By knowing this distance, which is the hypotenuseof the measurement triangle and by knowing the angle that the sensortransmits relative to the vertical axis of the vehicle 225, themeasurement triangle is determined and the pitching angle of the vehicle225 can be determined, assuming that the rear portion of the track orthe rear wheels are touching the ground, and the transmission anglecorrected to give an accurate velocity of the vehicle 225 independentlyof whether the vehicle 225 is traveling on level ground or a hill.

FIG. 44B illustrates the case of a counterclockwise pitch of the vehicleand where the vehicle is traveling up a hill. The distance X to thereflecting ground 227 is measured based on time-of-flight or phaserelationships as discussed above and the measurement triangledetermined. The base altitude of the triangle can be calculated as Xsin(θ) and the altitude as X cos(θ) and then the pitch angle φ fromtan(φ)=(X cos(θ)−h)/X sin(θ). The new transmission angle θ{grave over ()} then is θ+φ (for angles positive clockwise) permitting a correctedcalculation of the velocity of the vehicle 225.

In some cases, it is desirable to add a compass such as a fluxgatecompass or a Honeywell HMR3000 magnetic compass manufactured byHoneywell. This addition makes it possible to also accurately know theheading of the agricultural vehicle and can allow some locationinformation of where the vehicle is on the field. This then permitscontrol of product distribution by location on the field in a much lessexpensive, although not as accurate, method as the Beeline systemdiscussed above. The device is not shown in the figures but can be addedas a component on a printed circuit board that contains other relatedcomponents. In some cases, it would be mounted elsewhere to minimize theeffects of metallic parts of the vehicles.

Although pitch has been the main focus herein, a similar system can alsobe used to measure roll of the vehicle if it sends and receivesradiation to the side of the vehicle. When the compass as discussedabove and a dual system as described here is used, all of the angles,pitch, yaw and roll as well as the vehicle displacements (and thusvelocities and accelerations) can be determined and a sort of low costinertial measurement unit has been created.

The speed sensing system or sensors 231 and 232 in accordance with thepresent invention is shown installed on an agricultural vehicle 230 inFIG. 45. The sensors 231 and 232 are installed high on the tractor so asto keep them away from contaminating dirt and other contaminants. Thesensors 231 and 232 include transducers and are built such that theirtransducers emit signals along a respective path 233 and 234 that arenominally each at an angle θ with the ground of approximately 35degrees. The vehicle is moving at a velocity V.

The sensors 231,232 may be installed on a frame of the agriculturalvehicle 230 or on any of the parts of the agricultural vehicle 230 whichmay thus constitute the frame. That is, the agricultural vehicle 230 mayinclude a storage compartment for storing product to be distributed, aconveying or distribution mechanism for dispensing the product to afield and a driving mechanism for conveying the product from the storagecompartment to the distribution mechanism at a controlled rate. Thesensors 231,232 may be mounted on the storage compartment, on thedistribution mechanism and on the driving mechanism so that all of thesecould constitute the frame of the vehicle 230.

FIG. 46 shows a block diagram of an ultrasonic velocity and distance toground sensor 231,232 in accordance with one embodiment of theinvention. The microprocessor or DSP, the processor 235, generates a 40KHz continuous signal (SO) from source 236 that is gated for a burst bygate 237 controlled by the processor 235. The burst is typically 16cycles to allow the transducer 238 to reach full output in amplitude.The output of the gate 237 is fed through a signal splitter 239 to theSend/Receive transducer 238. The echo signal R from the ground isconverted to a voltage by the Send/Receive transducer 238. This R signalis sent to the signal splitter 239 and then to an amplifier 247. Theamplifier 247 drives a mixer 248 which mixes the LO (local oscillator)249. The output of the mixer 248 goes to a 455 KHz intermediatefrequency (IF) amplifier 250. The output of the IF amplifier 250 isconnected to a gated PLL (phase locked loop) 251. This loop 251 is gatedON for lock during the received echo time. The output of the PLL 251contains the IF frequency plus the Doppler frequency and is input to amultiplier 636.

The SO source 236 and the LO 249 are inputs to mixer 252, the referencechannel. The output of mixer 252 drives an IF amplifier 253. The outputof the amplifier 253 is provided to the multiplier 254 so that theoutput of the multiplier 254 is the product of IF1 and IF2. The outputof the multiplier 254 will be the sum and difference of the twofrequencies. The sum is then filtered out and the difference is theDoppler frequency. The Doppler frequency is measured by themicroprocessor or DSP 235 and is scaled to indicate the speed in MPH orother convenient units.

The wheel slip of the tractor can be calculated, if desired, by takingthe difference of the wheel speed and the ground speed by inputting thewheel speed into the to/from external equipment input of themicroprocessor or DSP 235.

The IF amplifiers 250,253 have a narrow bandwidth to filter out noisefrom the echo signal. The IF signal has more cycles than the burst bythe ratio of 455000/40000 or (11.375). This is important so that thereare sufficient cycles for the PLL 251 to lock onto. For a 16 cycle 40KHz burst, the number of cycles for PLL lock is 182 plus the ringingcycles. The PLL 251 filters out and averages the frequency of the echo.The PLL output runs at the frequency of IF1 until the next sample to thePLL 251.

A display 255 is provided coupled to the microprocessor or DSP 235 anddisplays the information obtained from the sensor.

The frequency of 455 KHz was selected because of the many ceramicfilters that are available for this frequency. The IF bandwidth isdetermined by the IF amplifiers 250, 253. The drive level for theSend/Receive transducer 238 is high to maintain the best signal to noiseratio. Other frequencies of the IF amplifiers 250, 253 can be used inaccordance with the invention.

The circuit can be made very compact by using surface mount components.All of the circuitry except the transducer drive operates at 3.3 volts.

The speed of sound varies significantly with air temperature and to alesser extent with altitude. Compensation for the temperature effect canbe achieved through measuring the ambient temperature via temperaturesensor 256 coupled to the microprocessor or DSP 235 and calculating thevalue of the sound velocity, C, as is well known in the art. Similarly,a barometric sensor can be used to measure the atmospheric pressure andits effect on the speed of sound can also be calculated as is also knownin the art. Alternately, a second ultrasonic transducer can be placedwithin the field of the first transducer but at a known displacement andthe speed of sound can be measured. The measured or calculated value forC would then be used in the calculations of equations (1) or (2).

Thermal gradients caused by the sun heating the ground can also have asignificant effect on the returned waveform. This problem has beensolved and is disclosed in U.S. Pat. No. 6,517,107.

Finally, wind can also affect the system by causing density changes inthe air and diffracting the sound waves in much the same way as thermalgradients. The techniques for solving the thermal gradient problem suchas through the use of a logarithmic compression circuit will alsocorrect for this effect. To the extent that wind shortens the time forthe waves to travel to the ground in one direction, it will lengthenthat time in the other direction and the two effects cancel. A strongside-wind can deflect the path of the sound waves and create anincreased path to and from the ground and thus introduce an error.Normally this is small since the wind velocity is normally smallcompared with the speed of sound. In rare cases where this effect needsto be considered, an anemometer can also be incorporated into thesystem, and coupled to the microprocessor or DSP 235, and the wind speedcan then be used to correct for the path length of the ultrasound waves.

6. Distance Measurement

What follows is a discussion of a particular distance measuring design.FIG. 47 shows a circuit 260 of a distance sensor which is also capableof use in any of the embodiments disclosed herein for determining thedistance between the sensor and an object. For example, with referenceto the embodiments disclosed in FIGS. 37-46, the sensor can be used as adistance-to-ground sensor for determining the distance between themounting location of the sensor and the ground.

The circuit 260 operates on a radar frequency and includes a transmitter261 and a receiver 262. The transmitter 261 includes a 77 GHz IMPATToscillator 263 providing a signal to an amplitude modulator 264. Theamplitude modulator 264 also receives a signal indicative of a selectedrange F1, F2 or F3 and then directs a transmission of a signal at aspecific frequency from a transmitting antenna 265. The receiver 262includes a receiving antenna 266 which receives a signal reflected froman object, wherein it is desired to determine the distance between thesensor (in which the transmitting antenna 265 and the receiving antenna266 are mounted) and the object.

Receiver 262 also includes a mixer 267 provided a signal by a localoscillator 268 which combines the signal from the receiving antenna 266and the local oscillator 268 and feeds the combined signal to an AGC IFamplifier 269. The amplifier 269 amplifies the signal and feeds theamplified signal to a demodulator 270, with a feedback loop being formedto the amplifier 269. Optionally, the receiver 262 can be set up with anAFC loop for auto-tuning purposes. In this case, the frequency drift ofthe IMPATT oscillator 263 will have little or no effect on the operationof the circuit 260.

The signal from the demodulator 270 and the selected frequency range arefed to a phase detector 271 which derives the distance between thesensor in which the circuit 260 is embodied and the object based on thedetected phase difference. From successive distance measurements, thevelocity of the object can be determined.

Advantages of the circuit 260 are that it is a relatively simple circuitwhich can be constructed from readily available electronic components.

7. Scanners

In addition to rotating and vibrating mirror-based and theacousto-optic-based scanners discussed above, a development by TexasInstruments called Digital Light Processing (DLP®) provides theopportunity for the control of a laser or other light beam for scanningwithin or outside on a vehicle that is unprecedented. Using DLPtechnology, a laser beam can be divided into up to 1 million separatelyaddressable and pointable beams each with a two-dimensional scanningangle of plus or minus 12 degrees and a scanning rate of 40 KHz. Usingthe systems described above for locating ROIs, a group of beams, witheach beam just below the eye safety limit, can be directed to monitor aparticular object in the field of view. Thus if 10,000 pixels areallocated for each object in the field of view, up to 100 such objectscan be tracked with high resolution. Similarly, for internal vehicleapplications, the occupants of all vehicle seating positions can bemonitored and tracked even during a vehicle crash with few pixelsmonitoring uninteresting objects. When the window or door is closing,for example, a group of pixels can be aimed at the window or door to seethat there are no arms, legs or fingers in the path of the closing dooror window. In fact, many of the monitoring operations discussed in U.S.Pat. No. 7,164,117 and its related applications, all of which areincorporated herein by reference, can be effectively accomplished withthe DLP technology (see dlp.com for a description of the technology).Although the scanning angles are limited to 24 degrees, this angle canbe arbitrarily increased using lenses and/or mirrors as is well known tothose skilled in the art. Many additional applications for DPLtechnology for automotive uses will now become evident to those skilledin the art because of the disclosure herein including vehicle-to-vehicleor vehicle-to-infrastructure communication systems using DLP principles.A key invention herein is thus any use of DLP technology on any vehiclefor scanning, pointing, imaging and communication applications.

8. Summary

8.1 Exterior Monitoring

There has thus been shown and described a monitoring system formonitoring the exterior of the vehicle using an optical system with oneor more CCD or CMOS arrays and other associated equipment which fulfillsall the objects and advantages sought after.

More particularly, described above is a method for determining theidentification and position of objects exterior to a vehicle whichcomprises transmitting optical waves into the space surrounding thevehicle from one or more locations, and comparing the images of theexterior of the vehicle with stored images of objects external to thevehicle to determine which of the stored images match most closely tothe images of such objects such that the identification of the objectsand their position is obtained based on data associated with the storedimages. The optical waves may be transmitted from transmitter/receiverassemblies positioned at one or more locations around the exterior ofthe vehicle such that each assembly is situated where it has a good viewof a particular space near the vehicle. Each assembly may comprise anoptical transmitter (such as an infrared LED, an infrared LED with adiverging lens, a laser with a diverging lens and a scanning laserassembly, an infrared floodlight, or other light source) and an opticalarray (such as a CCD array and a CMOS array). The optical transmittermay operate in the eye-safe part of the electromagnetic spectrum. Theoptical array is thus arranged to obtain the images of the exterior ofthe vehicle represented by a matrix of pixels. The output from eacharray can be compared with a series of stored arrays representingdifferent objects using optical correlation techniques. Preferably, alibrary of stored images is generated by positioning an object near thevehicle, transmitting optical waves toward the object from one or morelocations, obtaining images of the exterior of the vehicle, each from arespective location, associating the images with the identification andposition of the object, and repeating the positioning step, transmittingstep, image obtaining step and associating step for the same object indifferent positions and for different objects in different positions.This is similar to the training and adaptation process described in U.S.Pat. No. 6,529,809.

One of the advantages of the invention is that after the identificationsand positions of the objects are obtained, one or more systems in thevehicle may be affected based on the obtained identification andposition of at least one of the objects. Such systems include a visualand/or audio warning system to alert the driver to the presence,position and/or velocity of objects in the blind spots as well as asystem for adjusting the turning resistance of the steering wheel toprevent movement by the driver into the path of an object. Anothersystem could be associated with the turning indicators to provide analarm if a turning signal is activated when an object is present in theblind spot which would interfere with the intended turn.

The image comparison may entail inputting the images or a part or formthereof into a neural network, combination neural network, or otherpattern recognition system that provides for each image, an index of astored image that most closely matches the inputted image. The index isthus utilized to locate stored information from the matched imageincluding, inter alia, a locus of the center of the front of the objectand an appropriate icon for display purposes. To this end, a displaycould be provided in the passenger compartment or through the use of aheads-up display to provide a visual overview of the environmentsurrounding the vehicle. The icons could be general icons of objects ingeneral or more specific icons indicative of the type of vehicle, etc.Moreover, the position of the object relative to the vehicle may bedetermined so that an action by the driver of the vehicle that mightresult in an accident is prevented. It is also possible to obtaininformation about the location of the object from the image comparisonand adjust the position of one or more of the rear view mirrors based onthe location of the object. Also, the location of the object may beobtained such that an external light source may be directed toward theobject to permit a better identification thereof.

In addition, the location of the locus of the center of the objectexterior to the vehicle may be monitored by the image comparison and oneor more systems in the vehicle controlled based on changes in thelocation of the locus of the center of the object exterior to thevehicle over time. This monitoring may entail subtracting a mostrecently obtained image, or a part thereof, from an immediatelypreceding image, or a corresponding part thereof, and analyzing aleading edge of changes in the images or deriving a correlation functionwhich correlates the images with the object in an initial position withthe most recently obtained images.

In another method for determining the identification and position ofobjects external to the vehicle in accordance with the invention,optical waves are transmitted into a space near the vehicle from aplurality of locations, a plurality of images of the exterior of thevehicle are obtained, each from a respective location, athree-dimensional map of the exterior of the vehicle is created from theimages, and a pattern recognition technique is applied to the map inorder to determine the identification and position of the objects. Thepattern recognition technique may be a neural network, fuzzy logic or anoptical correlator or combinations thereof. The map may be obtained byutilizing a scanning laser radar system where the laser is operated in apulse mode or continuous modulated mode and determining the distancefrom the object being illuminated using time-of-flight, modulated wavesand phase measurements with or without range gating. (See for example,H. Kage, W. Freemen, Y Miyke, E. Funstsu, K. Tanaka, K. Kyuma“Artificial retina chips as on-chip image processors andgesture-oriented interfaces”, Optical Engineering, December 1999, Vol.38, Number 12, ISSN 0091-3286.)

In a method for tracking motion of a vehicle in accordance with theinvention disclosed above, optical waves are transmitted toward theobject from at least one location, a first image of a portion of thespace exterior of the vehicle is obtained, the first image beingrepresented by a matrix of pixels, and optical waves are transmittedtoward the object from the same location(s) at a subsequent time and anadditional image of the particular space exterior of the passengercompartment is obtained, the additional image being represented by amatrix of pixels. The additional image is subtracted from the firstimage to determine which pixels have changed in value. A leading edge ofthe changed pixels and a width of a field of the changed pixels isdetermined to thereby determine relative movement of the object from thetime between which the first and additional images were taken. The firstimage is replaced by the additional image and the steps of obtaining anadditional image and subtracting the additional image from the firstimage are repeated such that progressive relative motion of the objectis attained.

Also disclosed above is a method for controlling the steering system ofa vehicle which comprises transmitting optical waves (visible, IR oreye-safe IR) toward an object located in the vicinity of the vehicle,obtaining one or more images of an exterior space proximate to thevehicle, analyzing each image to determine the distance between theobject and the vehicle, and controlling steering system to prevent theoperator from causing a collision with the object based on thedetermined distance between the object and the vehicle. The image may beanalyzed by comparing the image of a portion of the exterior of thevehicle with stored images representing different arrangements ofobjects in the space proximate to the vehicle to determine which of thestored images match most closely to the image of the exterior of thevehicle, each stored image having associated data relating to thedistance between the object in the image and the vehicle. The imagecomparison step may entail inputting the image or a form or part thereofinto a neural network, combination neural network, system of neuralnetworks or other pattern recognition system that provides for each suchimage, an index of a stored image that most closely matches the image ofthe exterior of the vehicle. In a particularly advantageous embodiment,the size of the object is measured and a vehicle system is controlledbased on the determined distance between the object and the vehicle andthe measured size of the object.

In another method disclosed above for determining the identification andposition of objects proximate to a vehicle, one or more images of theexterior of the space proximate to the vehicle, or part thereof, ofradiation emanating from the objects proximate to the vehicle, and theimages of the radiation emanating from the objects are compared withstored images of radiation emanating from different objects proximate tothe vehicle to determine which of the stored images match most closelyto the images of the exterior objects of the vehicle such that theidentification of the objects and their position is obtained based ondata associated with the stored images. In this embodiment, there is noillumination of the object with optical waves. Nevertheless, the sameprocesses described above may be applied in conjunction with thismethod, e.g., affecting another system based on the position andidentification of the objects, a library of stored images generated,external light source filtering, noise filtering, occupant restraintsystem deployment control and the utilization of size of the object forvehicle system control.

Additionally disclosed above, among other things, is an arrangement forobtaining information about objects in an environment around a vehiclewhich comprises a light emitting device arranged on the vehicle foremitting infrared light into the environment around the vehicle, areceiver system arranged on the vehicle for receiving infrared lightfrom the environment around the vehicle and a measurement device coupledto the light emitting device and the receiver device for measuring timebetween emission of the infrared light by the light emitting device andreception of the infrared light by the receiver device. The measuredtime correlates to distance between the vehicle and an object from whichthe infrared light is reflected. The light emitting device may comprisean array of laser diodes, a pulsed laser or a continuous laser beamdirecting infrared light in a line and controls the laser beam to changea direction of the infrared light such that infrared light traverses avolume of space alongside the vehicle. In the latter case, the receiverdevice could comprise a single pixel receptor. Otherwise, the receiverdevice may comprise a CCD array, a CMOS array, an HDRC camera, a dynamicpixel camera and an active pixel camera.

A processor or control circuitry is usually coupled to the receiverdevice, by the use of wires or even wirelessly, for providing anidentification of the object from which light is reflected. Theprocessor preferably utilizes pattern recognition techniques such as aneural network or even possibly a modular neural network to determinethe distance between the vehicle and the object, the position of theobject and/or the identity of the object from which light is reflected.The processor can be designed to create a three-dimensional map of aportion of the environment surrounding the vehicle based on the receivedoptical waves or energy, and then extract features from thethree-dimensional map. In the latter case, a display is provided in thepassenger compartment visible to a driver of the vehicle for displayingfeatures or representations derived from features extracted from thethree-dimensional map.

As to the position of the light receiving components and associatedreceivers, they may be collocated or spaced apart from one another. Whenused for blind spot detection, they should be positioned around thevehicle to encompass the blind spots of the driver.

A system for controlling a vehicular system based on the presence of anobject in an environment around a vehicle comprises any of the foregoingconstructions of the arrangement for obtaining information about anobject in the environment surrounding the vehicle. A vehicular system isthen adapted to be controlled or adjusted upon the determination of thepresence of an object in the environment around the vehicle. To thisend, a processor is coupled to the arrangement and the vehicular systemfor obtaining the information about the object based at least on theinfrared light received by the receiver device and controlling thevehicular system based on the obtained information. The vehicular systemmay be a display visible to a driver of the vehicle for displayingfeatures or representations derived from features extracted from athree-dimensional map generated by the processor from the optical wavesor energy received by the receiver device. The vehicular system couldalso be a steering wheel having an adjustable turning resistance, whichis adjustable with a view toward avoiding accidents, and an audio alarmand a visual warning viewable by a driver of the vehicle.

A main feature of one embodiment of the invention is the combination ofan optical system with superimposed infrared patterns. In other words,the basic system is a passive optical system. Another feature is the useof a high dynamic range camera that can be used to get the image. Athird feature is to use modulated light or triangulation to determinethe distance to an object in the blind spot. Another feature of theinvention is to interpret and identify the image rather than justoffering it to the driver. Still another feature of the invention is toprovide methods of obtaining three-dimensional information about objectsin the vicinity of the vehicle by using modulated illumination and phaseinformation from the reflected light.

Note as stated at the beginning this application is one in a series ofapplications covering safety and other systems for vehicles and otheruses. The disclosure herein goes beyond that needed to support theclaims of the particular invention that is being claimed herein. This isnot to be construed that the inventor is thereby releasing the unclaimeddisclosure and subject matter into the public domain. Rather, it isintended that patent applications have been or will be filed to coverall of the subject matter disclosed above.

8.2 Anticipatory Sensors

In general terms, disclosed above is an inflator system for inflating anairbag which comprises a gas inflow mechanism for inflating the airbagwith gas, a vent system (if present) for controlling removal of gas fromthe airbag, a first anticipatory crash sensor for determining that acrash requiring deployment of the airbag will occur based on dataobtained prior to the crash and, upon the making of such adetermination, directing the gas inflow mechanism to inflate the airbag,and a second crash sensor for determining that a crash requiringdeployment of the airbag will occur or is occurring and, upon the makingof such a determination, controlling the vent system to enable theremoval of gas from the airbag whereby the pressure in the airbag ischanged by the removal of gas therefrom enabled by the vent system.

The gas inflow mechanism may be in the form of an inflator, which can bean aspirated inflator, which is activated to produce gas and release thegas through conduits into the interior of the airbag. The gas inflowmechanism can also be in the form of a tank of pressurized gas and avalve in a conduit leading from the tank to the interior of the airbagwhereby opening of the valve causes flow of gas from the tank into theairbag. Any other type of structure or method which serves to causeaccumulation of gas in the interior of the airbag can also be used asthe gas inflow mechanism in accordance with the invention. The gasinflow mechanism can also constitute multiple inflators which areindependently activated based on, the severity of the anticipated crash.In this case, one inflator would be activated for a minor or averagecrash whereas for a more severe crash, two or more inflators would beactivated thereby increasing the flow of gas into the airbag and theinflation rate and/or pressure therein. Each inflator could becontrolled by the same or a different crash sensor.

The vent system may be in the form of a variable outflow port or ventintegral with the airbag, e.g., a flap built in an exterior surface ofthe airbag and providing a regulatable conduit between the interior ofthe airbag and exterior of the airbag (regulatable both with respect tothe amount of gas flowing therethrough and/or the rate of gas flowingtherethrough). The vent system may also be in the form of a conduitleading from the interior of the airbag to the exterior of the airbagand having a regulatable valve in the conduit whereby regulated openingof the valve causes removal of gas from the interior of the airbag. Insome cases, notably most curtain airbags, a vent is not used and inothers the gas from the airbag is vented back through the inflatorassembly.

The airbag may be, but is not required to be, a side airbag arranged toinflate between the occupant and the side door. Regardless of thedirection of the crash which will causes deployment of the airbag, it isbeneficial to provide some form of an occupant displacement permittingsystem arranged in connection with the seat for permitting the occupantto be displaced away from the airbag mounting surface upon inflation ofthe airbag and thereby increase the space between the occupant and theairbag mounting surface. Thus, if the airbag is a side airbag mounted inthe side door, it is beneficial to enable displacement of the occupantaway from the side door. Such occupant displacement permitting orenabling system may be in the form of some structure which introducesslack into the seatbelt in conjunction with the deployment of the airbagor a mechanism by which the seat can be moved or is actually moved awayfrom the side door, e.g., tilted inward.

The airbag can also be arranged to inflate to protect a rear-seatedoccupant and to this end, would be arranged in a back portion of theseat, attached to the back portion of the seat and/or integral with theback portion of the seat. It can also be inflated to protect occupantsfrom striking each other by deploying the airbag between adjacent seats.

For any positioning and use, the airbag can be arranged in a housing ofan airbag module. The airbag module could extend substantially along avertical length of the back portion of the seat for a side airbag.

Another embodiment of the inflator system comprises an inflator forreleasing a gas into the at least one airbag, a first anticipatory crashsensor for determining that a crash requiring deployment of the airbagwill occur based on data obtained prior to the crash and, upon themaking of such a determination, triggering the inflator to release gasinto the airbag, and a second crash sensor for determining that a crashrequiring deployment of the airbag will occur or is occurring and, uponthe making of such a determination, changing the rate at which gasaccumulates in the airbag. To this end, the second crash sensor isstructured and arranged to control outflow of gas from the airbag.Outflow of gas from the airbag may be controlled via a variable outflowport.

A method for inflating an airbag comprises making a first determinationby means of an anticipatory crash sensor that a crash requiringdeployment of the airbag will occur based on data obtained prior to thecrash and, upon the making of such a determination, inflating theairbag, and making a second, separate determination by means of a secondcrash sensor that a crash requiring deployment of the airbag will occuror is occurring and, upon the making of such a determination, changingthe rate at which gas accumulates in the airbag. The rate at which gasaccumulates in the airbag may be changed by enabling and regulatingoutflow of gas from the airbag.

In accordance with another embodiment of the invention, a vehiclecomprises one or more inflatable airbags deployable outside of thevehicle, an anticipatory sensor system for assessing the probableseverity of an impact involving the vehicle based on data obtained priorto the impact and initiating inflation of the airbag(s) in the event animpact above a threshold severity is assessed, and an inflator coupledto the anticipatory sensor system and the airbag for inflating theairbag when initiated by the anticipatory sensor system. The airbag maybe housed in a module mounted along a side of the vehicle, in a sidedoor of the vehicle, at a front of the vehicle or at a rear of thevehicle.

The anticipatory sensor system may comprise at least one receiver forreceiving waves or energy and a pattern recognition system, e.g., datastorage medium embodying a pattern recognition algorithm or neuralnetwork, for analyzing the received waves or energy or datarepresentative of the received waves or energy to assess the probableseverity of the impact. The pattern recognition system may also bedesigned to identify an object from which the waves or energy have beenemitted or generated, in which case, the assessment of the probableseverity of the impact is at least partially based on the identificationof the object. The pattern recognition system can also be designed todetermine at least one property of an object from which the waves orenergy have been emitted or generated, in which case, the assessment ofthe probable severity of the impact is at least partially based on thedetermined at least one property of the object.

Also disclosed herein is an airbag passive restraint system forprotecting an occupant adjacent the door in a side impact whichcomprises an airbag arranged to inflate between the door and theoccupant and a side impact anticipatory sensor for determining that anaccident requiring deployment of the airbag is about to occur prior tothe accident. The sensor is arranged to receive waves generated by,modified by or reflected from an object about to impact the vehicleresulting in the accident and comprises an identifying and determiningdevice for identifying the object based on a pattern of the receivedwaves and determining whether the identified object will cause anaccident requiring deployment of the airbag. The system also includes aninflator coupled to the sensor for inflating the airbag if the sensordetermines that an accident requiring deployment of the airbag is aboutto occur. The identifying and determining device may comprise a neuralnetwork trained on data of possible patterns of received waves inconjunction with an identification of the object the received waves havebeen generated by, modified by or reflected from. In the alternative,the identifying and determining device may comprise a fuzzy logicalgorithm or a rule based pattern recognition algorithm. The sensor maybe arranged to receive electromagnetic waves or acoustic waves.

Another disclosed embodiment of a system for triggering deployment of anairbag passive restraint system in anticipation of an accident betweenthe vehicle and an object approaching the vehicle comprises atransmitter arranged on the vehicle for sending waves toward the object,a receiver system arranged on the vehicle for receiving modified orreflected waves from the object and producing a signal representative ofthe waves, an identifying and determining device for identifying theobject based on a pattern of the received waves and determining whetherthe identified object will cause an accident requiring deployment of thepassive restraint system and a triggering system responsive to theidentifying and determining for initiating deployment of the passiverestraint system if the identifying and determining device determinesthat an accident requiring deployment of the passive restraint system isabout to occur. The transmitter may be arranged to transmitelectromagnetic waves, such as radar waves, or ultrasonic waves. Theidentifying and determining device may comprise a neural network trainedon data of possible patterns of received waves in conjunction with anidentification of the object the received waves have been modified by orreflected from, a fuzzy logic algorithm or a rule based patternrecognition algorithm. The transmitter may also comprise a lasertransmitter and the receiver system may comprise one or more chargecoupled devices or CMOS sensing arrays

Still another disclosed embodiment of a system for triggering deploymentof an airbag passive restraint system in anticipation of an accidentbetween the vehicle and an object approaching the vehicle comprises areceiver system for receiving electromagnetic waves generated, reflectedor modified by the object, an identifying and determining device foridentifying the object based on a pattern of the received waves anddetermining whether the identified object will cause an accidentrequiring deployment of the passive restraint system and a triggeringsystem responsive to the identifying and determining device forinitiating deployment of the passive restraint system if the identifyingand determining device determines that an accident requiring deploymentof the passive restraint system is about to occur. The receiver systemmay be arranged to receive light waves or infrared waves. As in theembodiments discussed above, the identifying and determining device maycomprise a neural network trained on data of possible patterns ofreceived waves in conjunction with an identification of the object thereceived waves have been generated, reflected or modified by, a fuzzylogic algorithm or a rule based pattern recognition algorithm. Thereceiver system may comprise a charge-coupled device or CMOS sensingarray of a plurality of such components.

Also disclosed is a method for controlling deployment of a passiverestraint system in anticipation of an accident with an approachingobject which comprises mounting at least one receiver on the vehicle toreceive waves generated by, modified by or reflected from an objectexterior of the vehicle, conducting training identification tests on aplurality of different classes of objects likely to be involved in avehicular accident, each of the tests comprising receiving wavesgenerated by, modified by or reflected from the object by means of thereceiver(s) and associating an object class with data from each test,and generating an algorithm from the training test results, associatedobject classes and an indication as to whether deployment of the passiverestraint system is necessary such that the algorithm is able to processinformation from the received waves from the receiver(s), identify theclass of the object and determine whether deployment of the passiverestraint system is necessary. During operational use, a plurality ofwaves generated by, modified by or reflected off an object exterior ofthe vehicle are received by means of the receiver(s) and the algorithmis applied using the received waves as input to identify the objectexterior of the vehicle and determine whether deployment of the passiverestraint system is necessary. At least one transmitter may be mountedon the vehicle to transmit waves toward the object exterior of thevehicle such that the waves are reflected off or modified by the objectexterior of the vehicle and received by the receiver(s).

Any of the airbag passive restraint systems described herein may be usedin conjunction with the variable inflation rate inflator systemdescribed above, or with aspirated inflators, and may be used inconjunction with one another to optimize protection for the occupant.

Another possible method entails the use of an externally deployableairbag system for protecting the occupant in a side impact with animpacting object. This method would involve determining that a sideimpact requiring deployment of an airbag outside of the vehicle betweenthe side of the vehicle and the impacting object is required based ondata obtained prior to the crash, and then inflating the airbag in theevent a side impact requiring deployment of the airbag is detected.

Some embodiments of inventions disclosed above comprises an anticipatorycrash sensor arrangement which provides information about an object suchas a vehicle about to impact the resident vehicle, i.e., the vehicle inwhich the anticipatory crash sensor arrangement is situated, and causesinflation of one or more airbags. For example, internal and/or externalairbags might be deployed. One particular embodiment comprises ananticipatory sensor system which uses (i) a source of radiant energyeither originating from or reflected off of an object or vehicle whichis about to impact the side of a target vehicle, plus (ii) a patternrecognition system to analyze the radiant energy coming from thesoon-to-be impacting object or vehicle to (iii) assess the probableseverity of a pending accident and (iv) if appropriate, inflate anairbag prior to the impact so as to displace the occupant away from thepath of the impacting object or vehicle to create space required tocushion the occupant from an impact with the vehicle interior. Insteadof or in addition to inflation of an airbag in the interior of thevehicle, an airbag may be inflated exterior of the vehicle to resist theforce of the colliding object and thereby reduce the severity of thecollision.

Although the primary area of application of some embodiments ofinventions disclosed herein is for protection in side impacts,embodiments of the invention also provide added protection in frontalimpacts by reducing the incidence of injury to out-of-position occupantsby permitting a slower inflation of the airbag and displacing theoccupant away from the airbag prior to the impact. Additionally, it canprovide added protection in rear impacts by reducing the incidence ofinjury caused by impacts of the occupant's head with the headrest byprepositioning the headrest adjacent the head of the occupant based onan anticipatory rear impact sensor.

Also disclosed herein is a method for obtaining information aboutobjects in an environment around a vehicle which includes emittinginfrared or visible light from the vehicle into a portion of theenvironment around the vehicle, receiving infrared light from theportion of environment around the vehicle, measuring distance betweenthe vehicle and an object from which the infrared light is reflectedbased on the emission of the infrared light and reception of theinfrared light, determining an identification of the object from whichlight is reflected based at least in part on the received infraredlight, creating a three-dimensional representation of the portion of theenvironment around the vehicle from which infrared light is receivedbased on the measured distance and the determined identification of theobject, and displaying on a display visible to the driver iconsrepresentative of the objects and their position relative to the vehiclebased on the three-dimensional representation.

There are numerous ways to measure distance between the vehicle, i.e.,the location of the sensor at which the infrared light is received andpossibly emitted. These include using structured light, measuring timeof flight of the infrared light, modulating the infrared light andmeasuring the phase shift between the modulated and received infraredlight, emitting noise, pseudonoise or code modulated infrared light incombination with a correlation technique, focusing the received infraredlight, receiving infrared light at multiple locations orstereographically, range-gating the emitted and received infrared lightand using triangulation.

Identification of each object may be determined using a trained patternrecognition technique or a neural network. Also, if images of theenvironment around the vehicle are formed from the received infraredlight, then the determination of the identification of each object canbe based on analysis of the images and on the measured distance. Eachimage can be processed in combination with the distance between thevehicle and the object from which the infrared light is reflected todetermine the identification of the object.

A method for controlling a vehicular system based on the presence of anobject in an environment around a vehicle in accordance with theinvention includes emitting infrared light from the vehicle into aportion of the environment around the vehicle, receiving infrared lightfrom the portion of environment around the vehicle, measuring distancebetween the vehicle and an object from which the infrared light isreflected based on the emission of the infrared light and reception ofthe infrared light, determining the presence of and an identification ofthe object from which light is reflected based at least in part on thereceived infrared light, and controlling or adjusting a vehicular systembased on the determination of the presence of an object in theenvironment around the vehicle and the identification of the object andthe distance between the object and the vehicle.

In one embodiment, the velocity of the object is determined and thevehicular system is controlled or adjusted based on the determination ofthe presence of an object in the environment around the vehicle, theidentification of the object, the distance between the object and thevehicle and the velocity of the object.

Also, the expected future path of the vehicle may be monitored and awarning provided when the expected future path of the vehicle approacheswithin a threshold distance of an identified object or vice versa.

The same techniques for measuring distance between the vehicle and theobject described above can be applied in this method.

Another method for controlling a vehicular system based on the presenceof an object in an environment around a vehicle in accordance with theinvention includes emitting infrared light from the vehicle into aportion of the environment around the vehicle, receiving infrared lightfrom the portion of environment around the vehicle, determining theposition and velocity of an object in the environment around the vehicleand from which infrared light is reflected based on the emission of theinfrared light and reception of the infrared light, classifying theobject from which light is reflected based on the reception of theinfrared light, and controlling or adjusting a vehicular system based onthe classification of the object and the position and velocity of theobject. The same enhancements to the methods described above can beapplied in this method.

This invention is also a system to identify, locate and monitor objectsoutside of a motor vehicle, such as a tractor, bulldozer, automobile ortruck, by illuminating the objects outside of the vehicle withelectromagnetic or ultrasonic radiation, and if electromagnetic,preferably infrared radiation, or using radiation naturally emanatingfrom the object, or reflected from the environment and using one or morehorns, reflectors or lenses to focus images of the contents onto one ormore arrays of charge coupled devices (CCDs) or CMOS arrays, or on asingle detector. Outputs from the CCD or CMOS arrays or other detectormay be analyzed by appropriate computational systems employing trainedpattern recognition technologies, to classify, identify and/or locatethe external objects. In general, the information obtained by theidentification and monitoring system may be used to affect the operationof at least one other system in the vehicle.

In some embodiments of the invention, several CCD or CMOS arrays areplaced in such a manner that the position and the motion of an objecttoward the vehicle can be monitored as a transverse motion across thefield of the array. In this manner, the need to measure the distancefrom the array to the object is obviated. In other embodiments, a sourceof infrared light is a pulse modulated laser which permits an accuratemeasurement of the distance to the point of reflection through themeasurement of the time-of-flight of the radiation pulse which may usethe technique of range gating. In still other embodiments, a scanningarray of infrared LEDs are used to illuminate spots on the object insuch a manner that the location of the reflection of the spot in thefield of view provides information as to the location of the objectrelative to the vehicle through triangulation.

In some embodiments, the object being monitored is the ground.

In still other embodiments, a scanning laser diode or equivalent ismodulated and the distance determined by phase measurement with orwithout range gating. The light can also be modulated with more than onefrequency, or with a random, pseudo-random or other code to extend therange without a loss in range accuracy at the expense of slightly morecomplicated electronics and/or software.

In some applications, a trained pattern recognition system, such as aneural network or neural-fuzzy system, is used to identify the object inthe blind spot of the vehicle. In some of these cases, the patternrecognition system determines which of a library of images most closelymatches the object in the blind spot and thereby the location of theobject can be accurately estimated from the matched images and therelative size of the captured image thus removing the requirement forspecial lighting or other distance measuring systems.

When the wave transmitter is arranged to transmit electromagnetic wavesand the wave receiver to receive electromagnetic waves, the processormay be arranged to determine both the distance between a wavetransmission and reception point and the ground based on a phasedifference or time of flight between the transmitted and received wavesand the speed of travel of the vehicle on the ground based on theDoppler velocity. The wave receiver optionally includes a notch filterfor filtering light other than infrared light emitted by the wavetransmitter. Optionally, the wave receiver comprises a light valve.

In another embodiment of the invention, the vehicle comprises a waveemitting mechanism arranged on the vehicle for emitting waves at anangle toward the ground near the vehicle, a receiver mechanism arrangedon the vehicle for receiving waves reflected from the ground near thevehicle, a vehicular system adapted to be controlled or adjusted basedon the velocity of the vehicle relative to the ground, and a processorcoupled to the wave emitting mechanism, the receiver mechanism and thevehicular system for determining the velocity of the vehicle relative tothe ground and the distance to the ground based on the transmission ofwaves by the wave emitting mechanism and reception of waves by thereceiver mechanism. The processor uses the determined distance tocorrect the determined velocity of the vehicle relative to the groundand controlling the vehicular system based on the corrected velocity.The waves are transmitted by the wave emitting mechanism in pulses ormodulated such that the processor is able to determine the distancebetween a wave-transmission and reception point and the ground and thevelocity of the vehicle based on analysis of the waves transmitted bythe wave emitting mechanism and the waves received by the wave receivermechanism. The same features described for the vehicle above may be usedin this embodiment as well.

As stated at the beginning this application is one in a series ofapplications covering safety and other systems for vehicles and otheruses. The disclosure herein goes beyond that needed to support theclaims of the particular invention that is being claimed herein. This isnot to be construed that the inventor is thereby releasing the unclaimeddisclosure and subject matter into the public domain. Rather, it isintended that patent applications have been or will be filed to coverall of the subject matter disclosed above.

The inventions described above are, of course, susceptible to manyvariations, combinations of disclosed components, modifications andchanges, all of which are within the skill of those practicing the art.It should be understood that all such variations, modifications andchanges are within the spirit and scope of the inventions and of theappended claims. Similarly, it will be understood that applicant intendsto cover and claim all changes, modifications and variations of theexamples of the preferred embodiments of the invention herein disclosedfor the purpose of illustration which do not constitute departures fromthe spirit and scope of the present invention as claimed.

Although several preferred embodiments are illustrated and describedabove, there are possible combinations using other geometries, sensors,materials and different dimensions for the components that perform thesame functions. This invention is not limited to the above embodimentsand should be determined by the following claims.

1. A motor vehicle, comprising: a safety device deployable outside ofthe vehicle; an anticipatory sensor system for assessing the probableseverity of an impact involving the vehicle based on data obtained priorto the impact and initiating deployment of the safety device in theevent an impact above a threshold probable severity is assessed; and anactuator coupled to said anticipatory sensor system and said safetydevice for deploying said safety device when initiated by saidanticipatory sensor system, said anticipatory sensor system comprisingat least one receiver for receiving waves or energy and a patternrecognition system for analyzing the received waves or energy or datarepresentative of the received waves or energy to assess the probableseverity of the impact, said pattern recognition system being arrangedto ascertain the identity of an object from which the waves or energyhave been emitted, reflected or generated, the assessment of theprobable severity of the impact being at least partially based on theidentification of the object, said pattern recognition system comprisinga processor embodying a pattern recognition algorithm designed toprovide an output of one of a number of pre-determined identities of theobject.
 2. The vehicle of claim 1, wherein the ascertaining of theidentity of the object by said pattern recognition system comprisesdetermining whether the object is a human, pedestrian, animal orcyclist.
 3. The vehicle of claim 1, wherein said anticipatory sensorsystem further comprises a transmitter for transmitting waves or energyaway from the vehicle, said at least one receiver receiving waves orenergy transmitted by said transmitter which have been reflected fromobjects exterior of the vehicle, the received waves or energy beinganalyzed to enable determination of the identity of the object.
 4. Thevehicle of claim 3, wherein said transmitter is arranged to transmitinfrared waves.
 5. The vehicle of claim 4, wherein said transmitter isarranged to transmit infrared waves in an eye-safe range.
 6. The vehicleof claim 4, wherein said transmitter is controlled such that theintensity of transmitted infrared waves is determined at least in partby the distance to the reflecting object.
 7. The vehicle of claim 3,wherein said anticipatory sensor system is arranged to apply rangegating to select a range of an object to be identified.
 8. The vehicleof claim 1, wherein the vehicle has a front, a rear, a left side and aright side, wherein said safety device comprises an airbag, furthercomprising an airbag module for housing said airbag, said airbag modulebeing mounted in the front of the vehicle such that when deployed, saidairbag covers a portion of the vehicle hood.
 9. The vehicle of claim 1,wherein the vehicle has a front, a rear, a left side and a right side,said safety device comprises a net, further comprising a module forhousing said net, said module being mounted in the front of the vehicle,said pattern recognition system being arranged to ascertain the identityof the object as a pedestrian such that said anticipatory sensor systeminitiates said actuator to deploy said net to capture the pedestrian andhold the pedestrian onto the vehicle after the impact.
 10. The vehicleof claim 1, wherein when said pattern recognition system identifies theobject as a non-human animal, said anticipatory sensor system withholdsor modifies initiation of said actuator.
 11. The vehicle of claim 1,wherein said pattern recognition system is arranged to receive imagesfrom said at least one receiver and analyze images of the object inorder to ascertain the identity thereof.
 12. The vehicle of claim 11,wherein said pattern recognition system is arranged to perform imagesubtraction when analyzing the images.
 13. The vehicle of claim 12,wherein said anticipatory sensor system further comprises a transmitterfor transmitting waves or energy away from the vehicle, said at leastone receiver receiving waves or energy transmitted by said transmitterwhich have been reflected from objects exterior of the vehicle, thereceived waves or energy being analyzed to enable ascertaining theidentity of the object, said at least one receiver and said transmitterbeing controlled such that said at least one receiver receives a firstimage of the object illuminated by waves transmitted from the vehicleand a second image of the object which is not illuminated by wavestransmitted from the vehicle, said pattern recognition system analyzingthe first and second images in relation to one another.
 14. The vehicleof claim 1, wherein the vehicle has a front, a rear, a left side and aright side, wherein said safety device comprises an airbag, furthercomprising an airbag module for housing said airbag, said airbag modulebeing mounted in the rear of the vehicle such that said airbag deploysrearward.
 15. The vehicle of claim 1, wherein the vehicle has a front, arear, a left side and a right side, wherein said safety device comprisesan airbag, further comprising an airbag module for housing said airbag,said airbag module being mounted on the left or right side of thevehicle such that said airbag deploys laterally outward from the left orright side of the vehicle.
 16. A method for protecting an occupant of avehicle during an impact between the vehicle and an object, comprising:detecting an impending impact between the vehicle and the object priorto the impact; determining the probable severity of the impending impactprior to the impact; and initiating deployment of a safety deviceoutside of the vehicle between the vehicle and the object in the eventthe impending impact is determined to be above a threshold severity, thestep of determining the probable severity of the impact comprising:receiving waves or energy from the object, and ascertaining the identityof the object based on the received waves or energy by inputting thereceived waves or energy or data representative of the received waves orenergy into a pattern recognition algorithm which ascertains theidentity of the object based thereon, the pattern recognition algorithmbeing designed to provide an output of one of a number of pre-determinedidentities of the object.
 17. The method of claim 16, wherein thevehicle has a front, a rear, a left side and a right side, furthercomprising housing the safety device in the front of the vehicle. 18.The method of claim 16, wherein the safety device comprises a deployablenet, further comprising housing the deployable net in the front of thevehicle.
 19. The method of claim 18, further comprising positioning thenet to capture the object and hold the object onto the vehicle after theimpact.
 20. The method of claim 18, further comprising initiatingdeployment of the net only when the object is a human such that when theobject is a non-human animal, deployment of the net is not initiated.21. The method of claim 16, further comprising designing the patternrecognition algorithm to ascertain the identity of the object byclassifying the object as at least one of a human, pedestrian, animaland a cyclist.
 22. The method of claim 16, wherein the step of detectingan impending impact comprises transmitting waves away from the vehicle,receiving the transmitted waves after the waves have been reflected fromthe object and analyzing the received waves.
 23. The method of claim 22,wherein the transmitted waves are infrared waves.
 24. The method ofclaim 23, wherein the infrared waves are in an eye-safe range.
 25. Themethod of clam 22, further comprising determining an intensity of theinfrared waves to be transmitted based at least in part on a distance tothe reflecting object.
 26. The method of claim 16, wherein the safetydevice comprises a deployable airbag, further comprising arranging theairbag to cover a portion of a vehicle hood when deployed.
 27. Themethod of claim 16, wherein the vehicle has a front, a rear, a left sideand a right side, further comprising: housing the safety device in therear of the vehicle, the step of detecting an impending impact betweenthe vehicle and the object prior to the impact comprising detecting animpending impact into the rear of the vehicle, the step of initiatingdeployment of the safety device outside of the vehicle comprisingdeploying the safety device rearward from the rear of the vehicle. 28.The method of claim 16, wherein the vehicle has a front, a rear, a leftside and a right side, further comprising: housing the safety device inthe left or right side of the vehicle, the step of detecting animpending impact between the vehicle and the object prior to the impactcomprising detecting an impending impact into the left or right side ofthe vehicle, the step of initiating deployment of the safety deviceoutside of the vehicle comprising deploying the safety device laterallyoutward from the left or right side of the vehicle.